Lect2-3 Theory-Practice_UsabiilityStudies-Oct2014(1)

Copyright T. Komischke, H. Strub, T. Sachs, J. Chin 2014

Lect 13: Theory & Practice of Usability Studies

User Experience Design IIIJohn Chin, [email protected], 2014

Contact Information

Copyright T. Komischke, H. Strub, T. Sachs, 2014

An broad overview of context- Who, Anyone- When, Anytime- Where, Anywhere- Why, Motivations- What, TaskThe missing link - How Along with some depth on many topics

2


Doctor Who? Who is John Chin?Education

Ph.D. in Cognitive Psychology specializing in Human Factors Human-Computer Interaction

University of Maryland, College Park (advisor: Kent Norman)

Work Experience

Currently, User Experience Strategist @ Verizon Wireless User research, Writing UI requirements, Usability testing, Accessibility, Connected Car, Messaging, 3 patents pending

HiTech – Contractor Homeland Security, TSA Cognetics – Trizetto Group Avaya, Alcatel-Lucent, AT&T Bell Labs, Telcordia Lewis and Clark College

3


Enough about me, what about you?Who are you?

Education and Work experience Interests and specialty

Why are you here?

Goals and Objectives Career development, job?

How can I help?

Needs and Aspirations Networking

4


Bottom line goals… 1) Be neutral and stick to the facts 2) Advocate for your users, all of them

Recruit representative sample of eventual product/system users Participants – users testing the design/implementation Audience – observers (possible development team members) Report findings and possible solutions to readers or stakeholders

Gain better insights and understanding of the issues Hope to find possible solutions Few hard and fast answers: mainly tradeoffs

5

Planning a Test

Who: Low vision or Blind user Where: Standing on the sidewalk at a bus stop When: During rush hour in the morning What: Pedestrian getting directions to library Why: Pick up model created on 3D printer at library How: Mobile device navigation application

6

What are the research questions

Research questions are not tasks Key questions drive focus of the testing and tasks

Research: Can people successfully order products? Will customers be able to pay their credit card bill? Will radiologists be able to diagnose cases, without errors,

efficiently? Will consumers like this better than their current camcorder?

Tasks: Typically tasks should be frequent and/or important How long will it take customers to find the due date for their

current statement?

7


Where is project in the product life cycle? Early?

Are there requirements that need to be gathered? Or are requirements finalized?

What’s the shape of design concepts? Is this the next version or a new product? Some will say “too early for usability” since much is undefined

Some will say “Do a (user-centered) focus group instead” Not yet 100% clear what it does Bottom line: A lot is vague, testing can have big impact,

but make sure you’re exploring relevant issues

Middle of process? Requirements finalized Screens rough, or designed for usability Lots already determined: Use it. [or not, at your own risk] Bottom line: Testing can have moderate impact, Fast

turnaround important. Choose issues where you’ll be listened to.

8


Where is the project in the product life cycle? (continued…) Late?

UI is likely designed Possibly even coded

No leeway on requirements (functionality) Delivery possibly even scheduled What is the goal?

If the team will update the UI, could have a meaningful impact Fitting some request “It needs usability” No-win if in “police” role Stakeholder may be looking for “lipstick on the pig”, and endorsement

that “Usability says it’s okay” If it misses the mark, you’re in a no-win situation Big risk: once it’s delivered, product manager might not want to change

UI once existing customers invested time to learn it. Bottom line: be diplomatic & realistic. 1) Consider risks to

company if don’t fix enough, 2) what can be fixed later, and 3) process so won’t repeat this mess next release or product.

9


Where is the project in the product life cycle? I prefer early-middle, or early Beware being brought in late: a no-win game

Especially if project planned without user research. No resources to create prototype (and no time) No resources to incorporate results (or time) No one might care Even if you have a sponsor, you’re being set up for a battle Focus on your data. It’s a business decision re what to do.

Be careful – think high level I’ve been surprised many times – people and organizations that

seem to have mature processes “organizations” – since companies have silos: one or many

organizations in a company may be awesome. Be careful every time you meet a new one.

10


What do you know? 1) The human (in general) Topics include:

Cognitive processor (and memory) Perceptual systems Social interaction ( Emotion Other topics: Persuasion, attraction, etc.

If you have time to read, get applied books on these topics For topics you really like, get theoretical books; read

research papers. If you don’t already have a good book list, I’ll have

suggestions next week

11


What do you know? 2) Your business:

If a consultancy (or an experienced department) How does your firm approach problems? Why do customers choose it? Know previous reports – especially with your client Know how to leverage templates: research plan, test documents,

reports Practically: consider ways to promote future work for colleagues – both

your team and other teams [without compromising my integrity]

12


What do you know? 2) Your business

If you’re an employee: Get to know others in complementary departments

Market Research (make friends, which also reduces risk of being seen as competition)

Customer Service (Call Center “top 10” lists) Meet, go to lunch, read reports

VOC (Voice of the Customer) data Becoming increasingly popular Whatever you have available.

ForeSee:Survey at end of visits

13


What do you know? 2) Your business, if you’re an employee, continued

VOC data, continued OpinionLab

Input possible on any page Works best with meaningful URLs

Beware: Length. Tempting to continue asking questions, but not paying

customers Order matters: What goes early late, which questions go before others

14



Analytics (i.e., big data)

15

Copyright M. Tremaine, T. Sachs, A. Melewski 2012

This is a sample, not an Adobe endorsement



Analytics (i.e., big data) The ability to track… Beware: What the data makes easy, versus what is of interest

I have been able to track interactions on a page (sometimes every page), but cannot track a user through paths

Who are your company’s customers? Who use online systems? How do people access your stuff?

What are trends? Social networking analysis: As many as company follows Reports: JD Power, Forrester, Keynote, etc. Data collection services, and reports, (usually not cheap) Technology trends

The continuing rise of mobile, tablets The impact of OS, browsers, etc. Zeitgeist: e.g., Responsive Web Design

16


What will you test with? (1 of 3) Paper prototype?

Cheap, usually easy to create, conceptual Who will create? Will it be respected internally? “Photoshop” [i.e., online] paper prototype more “real” than paper Paper does communicate “conceptual”, room for change

I usually test paper as concepts that extend prototype(s)

17


What will you test with? (2 of 3) Visio? (or similar, PowerPoint can be fast)

I’m not a “designer” but I can design fairly well in Visio If my concepts, I’m in control: can test what I want (or edit myself) If my concepts, distracting from focusing on testing

Key point: Hyperlinks, so it looks like it’s functioning More about screens than functionality (won’t function well, i.e.,

fields cannot be filled in)

Something that really runs?: Html, iRise, Axure, Dreamweaver, Ruby, etc.

Functionality is great addition to realism, believability

18


What will you test with? (3 of 3) Possibly bad addition to cost, time, and pain, since

someone has to build & support the prototype. Arranging for someone to create the prototype can be hard, takes

time. Don’t assume getting resources will be fast/easy.

For many audiences, many products/systems, this is a necessity

Even more important for mobile UIs – I believe form factor really matters for mobile What do you think about where mobile is tested? [lab vs. field]

Trend of hybrid device approach: Big addition to complexity Prototype web, tablet, phone? iOS ? (pre-7 & 7?), Android (“flavor”?), Windows mobile,

Blackberry? App versus mobile web?

19


How will you record the test? Screen critical – see what happened Audio critical – hear what happened,

High quality audio often undervalued. Stereo audio recordings can be especially useful, since with

headphones you can pick out voices

Very useful: The participant’s face, to see emotions Still useful – hands on keyboard, on artifacts, etc.

Mobile technology especially challenging to record Harder to get good screen image Want to see what hands do, but hands/fingers cover the screen Still want to see face of the participant

20


How many participants to run? (1 of 3) Nielsen’s rules of thumb:

5 or a few more. Go higher if test has truly different groups

Consider more rounds of testing, if prototyping and time are available I often have trouble getting prototyping for multiple

rounds of testing) Bump up numbers a little if only one round of testing

21


How many participants to run? (3 of 3) Hank’s rules of thumb

Arranging a usability test is hard Unless really simple, run at least 3 days

60 minute sessions: 15 participants 90 minute sessions: 12 participants

Beware going > 4 days at a crack – exhausting, and too much to analyze Too much for industry people to pay attention to

If need more people, run another series Or outsource the usability

22


How many participants to run? (3 of 3) How many customer “demographic groups” to recruit

Rule of thumb: shoot for at least 3 people per demographic

If differing opinions, still have a tie-breaker If a no-show, still have 2 If it’s a key customer group (i.e., “prospects”) and site is

for them, increase or subdivide that group No point in testing 12 groups for study with 15 participants

23


How much time per session? Guesstimate: How big/complex does the UI feel? If testing in cafeteria (or some place public) keep sessions

really short To cover more ground, lots of sessions covering different tasks

In lab, I believe no shorter than an hour Lots of overhead, effort, to recruit, to prep, to get observers, to

analyze

I find it hard to go > 1-1/2 hours Sessions are tiring, for participant, moderator, observers Hard for many types of participants to get lots of time during

business hours – might get more strange people > 90 minutes, schedule in a break (which lowers efficiency)

I’ve seen 2 and 2-1/2 hour sessions (and no break) in last year – and they worked, but moderator knew domain well

24


When to schedule sessions? Easiest: during business hours

If live observers, much better to have straight days & business hours

But will you get enough of “right” people?

Who are target participants? What are their days like? Many jobs: work day Physicians: One early (before 9) one at lunch, 1-2 starting at 5 pm

Think about yourself. I rarely schedule before 10 in Manhattan since hard to get there and lots to get lab ready.

If won’t have live observers, can run any time, including a few per day at odd hours

For some groups, consider weekends But if you have family or other personal issues (like you have a

life), get more money to pay participants

25


When not to schedule sessions Consider observers as well as participants Holiday season is hard If winter, have backup plan If big storm might arrive during test, have backup plans Allow time for analysis, and for results to be incorporated

Often have designers responding before discussion.

26


How many sessions to run per day? Don’t make testing days too hard on anyone (including the

moderator) Want to be as alert, as good of a moderator for first and last

session Breaks between every session gives flexibility for buggy

prototypes, late participants, rebooting computer, trying something new, etc 60 minute sessions: up to 5 per day

Breaks of at least 30 minutes per session (even if just to allow for lateness), 60-90 minutes for lunch

90 minute sessions: up to 4 per day These are more tiring, make sure you have break time

Schedule lightly on first day of testing, to allow updating of everything – even your discussion guide, and to plan around unexpecteds When possible, run 1 or 2 participants first day, after lunch

27


How to recruit participants? (1 of 2) What are you testing? How specialized the community?

What’s your budget? How much time do you have? Many options, all involve tradeoffs

Professional recruiters: For many types of consumers, and some target audiences (like MDs) Most say they’ll recruit anybody for any reason Realistically, they have specialties Most have “lists” of known people, who may be semi-professional test

takers (some seem jaded) Recruiters aren’t cheap

Lists of customers, you contact yourself If you want to test improvements to Emerson’s HMI, want people who

know today’s Emerson HMI Beware: lists can take a LONG time to get (6 weeks or more)

Referrals from sales staff, or from technicians who know their “cool” and “boring” customers

Subscriber lists, Professional Society members (especially valuable if want special expertise)

28


How to recruit participants? (2 of 2) Recruiting from your web site

But any issue having “customer information”?

Waiting room at medical clinic Your card is given out by someone trustworthy (like a

doctor) “Mechanical Turk” at Amazon Craig’s List, or similar ads Your friends/acquaintances probably not diverse enough.

Beware effects of participants feeling obligated

Children can be especially challenging You want parents to be nearby so everyone is trusting, but you

have to keep them occupied

29


Who to recruit If clear that there’s one demographic (or “persona”), go for

it But consider going for sub-demographics as more participants

I usually shoot for up to 3, maybe 4 max, demographic groups. Product space may be much broader Target groups that are truly different, in parameters that matter

to your UI Many factors can be targeted independently, with same

participants. E.g.: Ranges for age, education, income Separate from experience in what you’re testing

Avoid trap of recruiting only the young/well-educated They look nice on video, have great things to say, but… Unless all users will be young and well-educated I learn a lot from diversity

30


Who to recruit: Common screening criteria (1 of 2) No one in your industry, or with a household member in

your industry No one in design or usability Always ask for a range, within whatever criteria.

If open to unemployed or retired, they’re probably easier to find for “business hours” studies than people who work, so “up to 2”

Internet skill: Want people who use web themselves, often, familiar

Age: cover diversity of your user community I’m pushing for older but not above 69 Many older users, but tend to talk more, and therefore get less

done (in my experience) I’ve had most dishonesty with elders: age (not be too old), internet skills

Consider gender distribution

31


Who to recruit: Common screening criteria (2 of 2) Education: I’ve known researchers who screen for minimum

of some college for general consumer products. Getting some without college is more honest test of usability, for

those with low literacy skills – who buy and use things I’m often fascinated with responses of those with least education

Ethnic diversity Income: As low as appropriate. If you want some truly

wealthy people, might need to specially recruit and pay more

But can ask for people whose speech is easy to understand (critical)

Where they live: City vs. suburb; and which city (Philly vs. NY)

32


Special participants: Children, Disabled, Employees Children:

Want a witness, at all times Parent should be nearby Make it fun Recruit so you’ll get interested kids

Disabled Hard enough to get disabled with full cognitive skill Check literature if recruiting disabled without full cognitive skill

Employees Is it their choice? Coerced? Impact of losing time on job Fear if criticize system, or people, when it’s recorded

33


Possible Forms for participants (1 of 2) Informed consent form has two goals:

Inform participants of what they’ll be doing, realistically, any risks (even of embarrassment), and benefits (often, interesting tasks; chance to help change a product; and get paid)

Consent: So it’s clear what they’re consenting to

Media Release: That you are recording them How recordings will be used (for study, nothing else) Can protect you as well as participants: if colleague loves a clip-

wants to show at a conference (“Sorry!”) If session is that good, can go back and pay more, or recreate the

scene with actors

34


Possible Forms for participants (2 of 2) NDA (Non-disclosure Agreement)

If you’re showing proprietary concepts Explain NDA too – so understand I’ve heard that a properly written and executed NDA protects

even if people talk – you did your due diligence

Liability release (?) In case incidentally hurt when coming to your facility

Include conditions that lawyers want But fight for plain language Long forms are harder to understand –consider more short

forms than fewer long forms

35


Don’t fight the law (but then I’m married to a lawyer) This is probably all new for most corporate council Draft pretty good sample forms before approaching

Shows you know something Easier to edit than start from scratch

Be able to explain what’s there, why, who and what your intentions are

Provide articles on what you’re after, or books Negotiate, but with the company’s best interests in mind

36


How much to pay participants? Goldilocks rule of thumb:

Not too much, not to little

If doing internal testing on employees, might be limited by corporate rules (participants already missing work during work time

For external participants, ask your recruiter for advice Consider adding in reasonable extra payment for

transportation/parking costs BTW, if paying a participant more than $500 in a year,

check with company HR about possible need to report “salary” to IRS Only an issue if have same person return

37


Should I use floaters? Floaters are backup participants, usually recruited to cover

two sessions Extremely valuable in case of no-show Also, for when a participant is totally inappropriate But, who has time to hang out for 2 sessions? Be careful

I use best recruiter I can, request to go for 100% turnout I’ve had odd birds as floaters

38


How to record a test? Morae, by Techsmith: The “Honda Fit” of usability

recording: $1500 for a full license Records two streams, usually screen and webcam on face, as well

as an audio stream If built-in mic isn’t awesome, get a good external mic

1copy of Morae includes 1 copy of “Observer”, which lets people in another room watch live (delayed ~20 seconds)

Very reliable Not bad to learn, to set up Can take notes, and make marks, live (but I don’t see that

done) Will make highlights videos Pretty efficient storage, so many sessions can go on a DVD

39


Recording a test – More on Morae Morae also good for portable lab – only need 2 laptops Requirements for computers not clear: doesn’t reflect multi-

core processors In my experience, Morae has run fine on machines well

below the “minimum recommended” capability

A hack to get 3 images is a hardware video mixer, that will superimpose for you.

40


Recording a test: Lots of fancier software, but.. Pricier, need fancier computer and equipment, can be

testier I know many big companies use OvoStudios My personal experience: very nice & helpful, not as reliable

as I’d like But I’ve heard of companies that find it trustworthy Network to find something that works for your needs.

Consider: HD looks great, awesome detail Probably not needed for routine recordings of basic web sites

41


Recording a test: Back to mobile How to get a good screen? Apple to HDMI to DVI cable, or

Apple TV hack if security allows License supports 3 images – I think all 3 are important [Note: target area – ask participants to keep the phone

THERE

42


Other Technology Issues: Eye Tracking: compelling to watch, help observers stay

awake Beware of heat maps: If interrupt participants, etc. Magic is uninterrupted, enough people to get good trends, so

software can analyze what happened Web site also needs to support eye tracking: overlays, modals

mess things up Software only knows the URL, not what’s on screen

Emotion tracking: GSR (how emotional) and valence (+/-) Affectiva is a company to watch

43


The Moderator’s Guide (1 of 4) Introduction Why you’re there That this is usability so

A) Please speak aloud (we’ll practice), B) Testing the UI, not testing you, C) Don’t worry about errors since prototype (not real data or

system)

Ask questions (though I might not answer) Can stop at any time, for any reason

Pay at beginning of session: Show trust (people don’t walk away)

Plan the task list Based on the research questions Ideally, based on your knowledge:

What are the true top tasks (consider what Gartner, JD Power say)

44


Moderator’s Guide (2 of 4) Think about time. Consider optional tasks to fill sessions for

fast participants. Or have an extra task or two, and prioritize list in pilot sessions

Moderator’s guide: By each task I like to have questions to probe, and suggested things to observe Participant’s guide just has the tasks – other stuff is deleted Edit moderator’s guide, either regenerate Participant’s guide, or

be careful on version control.

Questions to ask: Per task: Make sure to ask the goal of the task

Find a value? Perform a task Otherwise, think about what you’ll learn

Can ask questions like “How quickly do you expect to do this task? How easy do you expect this task to be?

After: How easy was it to perform this task? How often do you perform this task?

45


Moderator’s Guide (3 of 4) Questions to ask: At end of Session

I like to ask: What did you like best about the prototype system? What did you like least about the prototype system? [Sometimes in form of “What were 3 best & 3 worst things?”]

The SUS is trendy [SUS = System Usability Scale] Original by John Brooke, 1986, DEC

46


47The original SUS


Moderator’s Guide (4 of 4) Thoughts on SUS:

Alternates positive and negative questions, so respondent needs to be awake

Around since the 80’s, lots of experience in the industry A few common updates [i.e., “website” instead of “system”] Like school grading: 70 is okay, 80 good, 90 superb But maybe UI’s are better these days -> I’ve seen mainly upper

80’s or better

Net Promoter Score (NPS): How likely are you to recommend this product/company to a friend? Trendy, originally published in Harvard B-School Journal

Were participants supposed to learn something? Though be careful about testing learning in 1-hour long session,

unless that’s what system is for 1 hr not much time to learn, especially when many tasks

48


Cautions on locations (1 of 2) Dedicated lab:

Will you use it enough? Perhaps try lending it out, but beware… Sufficient staff support? [reception & waiting area, space for

observers who have meetings, snacks & drinks and meals] Window: Great, but then keep observer room quiet and dark Need to keep equipment secure

Conference rooms and a portable Morae lab Still have to keep laptops secure Will you always get good rooms for test when you need them?

Others’ office Conference rooms: Will your Morae computers run on their

network? If not, how will they communicate? Fairly safe from interruptions, though.

Cubes: Beware interruptions… Phone will ring & cube’nik might need to pick up; Co-workers don’t recognize you as important

Offices: Probably okay as long as participant respects you

49


Cautions on locations (2 of 2) Remote Testing

Success depends on installing web-sharing software, but software, computers and bandwidth are all pretty good Usually only 5 minutes, sometimes much worse

Better with web cam Surprisingly to me, rarely interrupted, but phone calls seem to be

respected

3rd party facility (i.e., market research company) Surprisingly pricey – be prepared for unexpected costs But they handle the details – REALLY nice When I use a facility, I have them recruit – so one point of failure

Even professionals sometimes use cafeteria, or Starbucks Best for quick “first impression” studies, lots of people in little

time Can recruit for more in-depth study (follow-up in lab…)

50


Screen-sharing software, for remote live observers Easiest to use the corporate standard

It’s IT-endorsed No firewall issues Account likely easy to set up, and/or use If there’s a problem, you used what company wanted

Otherwise, I believe in WebEx & a telephone bridge Usually reasonably pain-free to install Performance (delay, quality, frame-rate) pretty good Not cheapest, but trustworthy is important to me

51


Running a Test52


Be like Johnny Carson(Used to host The Tonight Show) Always genial, friendly Show was about the guest(s), not him THEIR thoughts & opinions Johnny didn’t lead But he got guests to talk When there was a problem, he took the blame (never the

guest’s fault unless something terrible) Was incredible at improvising

For more on Johnny, watch for rebroadcast of PBS documentary

53


Taking notes during: If you follow template strictly and have time, excel template

for all key questions and observation points is neat Time consuming to set up Not convenient to use when with participant

Can take notes with computer Not best for establishing rapport

Take notes at a constant pace (lots of them) So don’t communicate what you care about So writing when participant thinks they’ve said something

important

Compatriot’s notes (ideally, not in sight of participant) can be great, but everyone looks for different things Who will be lead on the report?

54


Things to watch during session55


What to do if: (1 of 2) Participant is no-show

Ask to arrive 15 minutes early: time for forms, for bathroom, and to call recruiter before session time

Call recruiter – check on reminders Backup slots available?

If remote, can’t get to internet Common to get weird firewall issues Have a local copy

Prototype fails Computer fails Participant arrives quite late Running really behind. Participant’s a talker, trouble, started

late Participant wants bathroom break, but already behind

56


What to do if: (2 of 2) Early in session, can see that participant doesn’t fit the

study? It appears that participant lied to recruiter re something

key? (After all, these studies pay well and economy is bad) You give cash or check to participant at start of session, and

he/she gets up and leaves? Participant admits to having committed a crime on your

video recording? (“When I do my taxes, I don’t put in all the receipts”)

You have strong evidence that participant is victim of or commits abuse, or is considering suicide?

57


After a test Debriefs are great. Consider recording them if you don’t

take good notes. Write down first impressions of results while they’re fresh in

your mind. Get invoices in promptly: * participants *, recruiter, and any

vendor you might use again Return borrowed equipment promptly (before damaged,

lost) Listen to sessions as “background music”. You will hear

things Push self for rapid turn-around.

Consider rehash/highlights meeting immediately afterwards, recorded, to get impressions of observers

Tight deadline for report doesn’t allow forgetting But beware getting slammed if rest of your job fell behind due to time

away for testing

58

Copyright T. Komischke, H. Strub, T. Sachs, J. Chin 2014

Questions?

Documents

Lect2-3 Theory-Practice_UsabiilityStudies-Oct2014(1)