Embed Size (px)
DESCRIPTION
Citation preview
Remote Research
Workshop
Photo Credit: recados-orkut.net
Remote Design Research
Definition:
Research where the participant is not sharing physical space with the researcher.
In person
Remote
% of qualitative research done remotely*
* Source: My wild guess
#UXLX #RR
Bayesian methodsAnimated pedagogical agency1995Infer when users needed helpDecision Theory & Adaptive Systems Group
Misunderstood animated pedagogical agent or spawn of Satan? ”
“
Bad Research
1) Academic Focus2) Lab-based3) False Premise
THAT IS WHY WE WENT REMOTE
Btw I’m Nate Bolt
12 YEARS293 STUDIES3,108 PARTICIPANTS (MODERATED)118 ORGANIZATIONS21,610 HOURS
“SOCIAL IMPACT OF DIGITAL TECHNOLOGY”
A DINOSAUR FAMILY EXPLAINS INFORMATION ARCHITECTURE
SF to PARIS IN TWO MINUTES
jumboltron.com
OMG PLEASE BUY THIS BOOK
Let’s jump right in
What’s New with UX Research Tools?
BUT FIRST
Moderated versus Automated
Moderated versus Automated
90%10%
Moderated versus Automated
Photo Credit: singularityhub.com
Integrated RecordingThat’s really awesome
GoToMeeting 5.2Moderated
HuggingModerated
Photo Credit: Diane Loviglio, Mozilla
At Mozillaand MailChimp
UsabillaAutomated
Mobile
HOW!TO!BEgraceful
When All Your TECHNOLOGYBREAKS
Follow the buddy system
It’s laptop time
remoteresear.ch
remoteresear.ch/lisbon
(BROWSER TIME)
IN CASE THE INTERNET FAILS
Research experience
Most important thing to get out of today?
Biggest fear or skepticism about remote research?
Thanks, Volunteer!
Four Takeaways
1. See moderated method & recording with GoToMeeting
2.Automated with Usabilla3.Self-moderated tablet hugging4.Quick incentive payment
Eff you sales guy
WHAT WE’LL COVER TODAY Recruiting
Moderated
Contextual Inquiry
Automated
Results
Pitfalls
Explode
SOME THOUGHTS ON WEB ANALYTICS
What do helicopters do in action movies?
How much practical research experience do I need to have for this workshop? ”
- Many People
“
Will this be a long lecture or an actual workshop? ”
- Some Guy on Twitter
“
Can we do contextual inquiries remotely? ”
- Nick Dunlavey | @ndunlavey
“
The physical and computing ecosystem
Now we do a little experiment
EXERCISE #1“Giraffe”
Binder PDF Page 4
Table groups
Pick one moderator and one participant
(they need to be in different locations)
10 minutes of moderating
remoteresear.ch/lisbon“The Workshop Binder”
MODERATOR Instructions
Your participant has a sheet of paper and that’s it
Goal: find out what kind of physical task instructions online
Hint: ask them to give you a play by play of what they are doing
Try and follow along with your own paper
NO helping, even if you could.
It’s A RACE
You only have 10 minutes to do this max.
Surprises?
Was there emotion?
WHEN DOES REMOTE WIN?
Marv.Creative director UX designer product manager developer IA co-founder.
Marv’s Boss.He’s always on a plane and loves birds.
Hey Marv, I want to save some money on research.
NATIVE ENVIRONMENTCHEAP ETHNOGRAPHY, NOT CHEAP LAB.
TIME-AWARERESEARCH IS THE BEST
I’LL TRADE YOU TIME FOR PHYSICAL PRESENCE
INVOLVED OBSERVERS
Easier to collaborate in real time.
TITLE TEXT
GEOGRAPHY
WHEN DOES REMOTE LOSE?
Hey Marv, I want feedback on our MMO video game from 13yo English speaking kids in France & Latin America with webcams.
I guess, Boss.
Might be hard to do with remote testing.
BANDWIDTHYes, it’s an issue.
TRANSLATIONIt’s expensive and difficult to do live.
MINORSConsent is a bear.
Especially if they’re under 13.
FACES
Most users don’t have webcams.
Webcams are a little invasive.
YOUR BOSS
SUPER DUPER SECURITY
AN EPIC FAIL.
Bad ResearchWe basically did
Kids will lie for coins
People mis-assessed their English levels
Bandwidth was really a problem
Kids don’t play when their parents are home
“ What happens when you let go of a balloon? ”
HOW TO FIND ACTUAL SPEAKERS OF YOUR LANGUAGE IN ANY COUNTRY:
“ Qu'est-ce qui se passe quand vous lâchez un ballon? ”
DESIGNINGREMOTE STUDIES
Yo Marv, get me some insight on WHY people are bailing out of checkout.
Sure, Boss.
Let’s combine methods. Intercept & observe 10 users live, and do 10 self-moderated with usertesting.com
TIME-AWARE TASKSTold to find purple pumps.
Doesn’t care.
Has to wear this for sister’s wedding this weekend.
Whole different interview.
HEART & SOUL OF REMOTE RESEARCH
Not scheduling participants.
REAL TIME OBSERVERS
TITLE TEXT
PROTOTYPES, WIREFRAMES, SKETCHES?
HECK YES.
RECRUITINGFOR REMOTE
RECRUITING OPTIONS
Ethnio
Links or house ads to online forms
Recruiting agencies
Customer email lists
Panels
Craigslist/online ads
RECRUITING OPTIONS
Grab someone in the hallway
Grab someone who isn’t directly involved in your project
IM your friends for a quick look
Panels and recruiting agencies
Real customers from marketing lists
Site intercepts
realness
SITE ISN’T AN OPTION
GEO IP
MEMBERS ONLY
GOLD STANDARD
Live recruiting
Get as close as you can
USE ETHNIO WHEN
Website access
Some budget
Quotas are relatively simple
USE FORMS WHEN
You can place a link but not code
Low budget
Quotas are relatively complex
Options:Google Docs
Wufoo
USE AGENCIES WHEN
No website access
People whine about first-time visitors
Higher budget
Online recruiting failed
CUSTOMER LISTS
Sub-segments
Ownership requirements
No website access
PANEL
SORT!OF
CRAIGSLIST
When other methods don’t work
No dough
Takes a while
Embarrassing to admit to your friends
LIVE RECRUITING MATH
Target: 6 recruits/hour
Allow for: 1% response (low average
You Need: 10,000 uniques/day
TITLE TEXT
remoteresear.ch/samples“Online Consent”
Paying Incentives
Collect minimal personal infoDeliver quicklyAppropriate for location
Screener constraints
Keep it short
Different from surveys
Who gets paid?
MODERATED must-haves
Consent
Tech set-up
The beef
De-installation
Thanks & incentive
AUTOMATED must-haves
Expectation setting
Instructions
The beef
Thanks & incentive
Recruiting
Same: Good “talkers”
Same: Demographic fits
Different: Relevant current
tasks
Different: Compatible tech
Logistics: What & How
Provide test target
Access
Credentials
Logistics: Who & When
How to connect with the user
Managing time zones
Session length
Paying Incentives
Collect minimal personal info
Deliver quickly
Appropriate for location
STAGED Incentives
For Contextual Inquiry
Start with $75, then at the end offer $25 more for environment photos
Self-made diary studies
Advanced planning:hybrid studies
Why use only one method?
Some methods don’t tell you the why
3-5 remote interviews is all you need
EXERCISE #2
Binder PDF Page 6
Use ethn.io or dev.ethn.io (beta) or wufoo
Under 8 Questions
Advanced Bonus: create an automated and moderated screener
okay, draft a screener
remoteresear.ch/lisbon“The Workshop Binder”
Marv-o - what in the world does automated/unmoderated/asynchronous research mean?
Sure, boss. There is an AVALANCHE of UX TOOLS.
“At least a little behavioral”
No surveys. No standard web analytics.
Since 2007
Can tools make us better researchers?
CATEGORIES
5
Conceptual
For A Long Time
Self-Moderated
Moderated
Web Analytics
Automated Static
Automated Live
Tasks On Concepts
For A Long Time
People Talk To A Machine
Talk To People
Fancy-Ass Analytics
Tasks On Images
Tasks On A Live Site
I.A. & Navigation
For Ad Agencies
Ammunition
Insight
Just Better Analytics
Still More Ammunition
More Ammunition
Conceptual
For A Long Time
Self-Moderated
Moderated
Web Analytics
Automated Static
Automated Live
Deep Behavioral UX
Insight
Opinions
Conceptual
For A Long Time
Self-Moderated
Moderated
Web Analytics
Automated Static
Automated Live
Ad Agencies Love It
Conceptual
For A Long Time
Self-Moderated
Moderated
Web Analytics
Automated Static
Automated Live
Jared Spool Hates It
Conceptual
For A Long Time
Self-Moderated
Moderated
Web Analytics
Automated Static
Automated Live
Mobile
Conceptual
For A Long Time
Self-Moderated
Moderated
Automated Static
Automated Live }
Remote Research ToolkitUNMODERATED (AKA AUTOMATED)
3 parts
Provide test target
Record user behavior
Capture comments (optional)
USABILLA
Clicks & QuestionsHeatmapsYou do your own recruiting$49 - $199
TITLE TEXT
LOOP11 (UN-MODERATED)
TITLE TEXT
GOTOMEETING - $39/MO
TITLE TEXT
WHAT WE USE. A LOT.
• Ethnio + GTM
• Usabilla + Usertesting.com
• Guerrilla mobile
EXERCISE #3
usertesting.com - 1 free user CODE: uxlisbon2012usabilla - 2 months free (Sign up for an account, but cancel on the payment page. Send an email to [email protected] and mention UXLX)
ethnio - 1 month free or trial. CODE: WORKSHOPLOVE
Either “unmoderated” or “practice interviewing”
RESULTS
REMEMBER, BEgraceful
When All YourTECHNOLOGYBREAKS
WHAT THEY WORRY ABOUT
WHO ARE YOU AND CAN I TRUST YOU?
IF YOU CAN SEE MY SCREEN,
CAN YOU SEE MY FACE?
WILL YOU BE ABLE TO TAKE
CONTROL OF MY COMPUTER?
WILL YOU HELP ME GET THE PLUG-IN
OFF MY COMPUTER AT THE END?
WILL THIS WORK IF I HAVE AFIREWALL/PROXY/STRICT IT
DEPARTMENT?
IS THIS GOING TO END UP ON
YOUTUBE?
OXYMORON?REMOTE EMPATHY
Taking Notes
Second monitor if possible
Excel Hack (remoteresear.ch/samples)
Observer Involvement
IM
Campfire (!)
Integrated Chat
Passing Notes
HYBRID STUDIES
Numbers actually do lie sometimes
And they don’t tell you the why
3-5 remote interviews is all you need
So, what could go wrong?
3 points of failure
Technologyyours
participant’s
observer’s
Recruiting
Test Design
Control what you can control
Which is, your set up.
You wouldn’t go backpacking without checking your gear.
Do it 24 hours in advance. Or else.
Prepare for whatyou can’t control
User’s dog trips over power cord
Internet affected by local weather
User can’t install
Doesn’t know how to install
Mid-SessionTech Problems
Live Recruiting Problems
What if they gave a study and nobody came?
Increase exposure
When you’re not getting enough views
When the wrong type of users are responding
Increase incentives
When it’s cheaper
When you’re almost getting
enough
When segments are tough
Switch methodsWhen base traffic level isn’t high enough
Script ProblemsCut the BS.
ADVANCED TECHNIQUESOne to many moderation
Multi-threading
PORTABLE RESEARCH
BURN THE SCRIPTFIND OUT WHAT MATTERS.
ADVANCED TECHNIQUESHigh Emotions
TITLE TEXT
THE FUTUREMOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE
DUAL LENSSTREAMINGFIELD RESEARCH
• Livestream• Flip Cam• It’s the view
finder, silly• Actual duct
tape
DISCOUNTS!!Usertesting.com 1 free user. uxlisbon2012
Usabilla 2 months free of any plan. Sign up for an account, but cancel on the payment page. Send an email to [email protected] and mention UXLX
Ethnio 1 month free any plan. WORKSHOPLOVE
remoteresear.ch/lisbon
(look in the slides)
DISCOUNTS!!
(for, well, that last slide)
