Remote Research Workshop, UX Week 2012 - Cyd Harrell

Remote ResearchWorkshop

Photo Credit: kaboodle.com

Remote Design Research

Definition:

Research where the participant is not sharing physical space with the researcher.

% of qualitative research done remotely*

* Source: My wild guess

In personRemote (2008)Remote (2012)

But it is pretty useful...in certain situations

THAT IS WHY WE GO REMOTE

#UXWEEK #RR

Btw I’m Cyd Harrell

and assisting me is Sarah Spitler

6 YEARS187 REMOTE!STUDIES

FAVORITE TOOLS: EMPATHY & DUCT!TAPE

(R.I.P.)

OXYMORON?REMOTE EMPATHY

DUAL LENSSTREAMINGFIELD RESEARCH

• Livestream• Flip Cam• It’s the view

finder, silly• Actual duct

tape

Since 2007

Can tools make us better researchers?

You can set up a pretty sweet rig

But if you’re not asking the right questions it doesn’t matter.

This is a great book by my old boss.

Let’s jump right in

What’s New with UX Research Tools?

A DISTINCTION

Moderated versus Automated


90%10%


Photo Credit: singularityhub.com

Integrated RecordingThat’s really awesome

GoToMeeting 5.2Moderated

Awesome compared to this!

HuggingModerated

Photo Credit: Diane Loviglio, Mozilla

At Mozillaand MailChimp

MobilePicture in picture

RecordingNice face, earthling

Mobile AppAutomated

GestureTracking

UsabillaAutomated

Mobile

Experience SamplingTask-driven

iOS & Android

Pix or it didn’t happen!

StorytellingiOS

Beyond Usability

HOW!TO!BEgraceful

When All Your

TECHNOLOGY

BREAKS

Follow the buddy system

It’s laptop time

remoteresear.chor dev.ethn.io/18014

(BROWSER TIME)

IN CASE THE INTERNET FAILS

Research experience

Most important thing to get out of today?

Biggest fear or skepticism about remote research?

bolt.ps

PRO TIP #1: MAKE A LANDING PAGE

Thanks, Volunteer!

No sales

HOW!TO!BEgraceful

When All Your

TECHNOLOGY

BREAKS

WHAT WE’LL COVER TODAY Recruiting

Moderated

Automated

Results

Pitfalls

Explode

SOME THOUGHTS ON WEB ANALYTICS

What do helicopters do in action movies?

Now we do a little experiment

EXERCISE #1“Giraffe”

Binder PDF Page 4

Table groups

Pick one moderator and one participant

(they need to be in different locations)

10 minutes of moderating

MODERATOR Instructions

Your participant has a sheet of paper and that’s it

Goal: find out what kind of physical task instructions online

Hint: ask them to give you a play by play of what they are doing

Try and follow along with your own paper

NO helping, even if you could.

It’s A RACE

You only have 10 minutes to do this max.

Surprises?

Was there emotion?

OXYMORON?REMOTE EMPATHY

Taking Notes

Second monitor if possible

Excel Hack (remoteresear.ch/samples)

Observer Involvement

IM

Campfire (!)

Integrated Chat

Passing Notes

WHEN DOES REMOTE WIN?

NATIVE ENVIRONMENTCHEAP ETHNOGRAPHY, NOT CHEAP LAB.

TIME-AWARERESEARCH IS THE BEST

I’LL TRADE YOU TIME FOR PHYSICAL PRESENCE

INVOLVED OBSERVERS

Easier to collaborate in real time.

TITLE TEXT

GEOGRAPHY

WHEN DOES REMOTE LOSE?

BANDWIDTHYes, it’s still an issue.

TRANSLATIONIt’s expensive and difficult to do live.

MINORS

Consent is a bear.

Especially if they’re under 13.

FACES

Most users don’t have webcams.

Webcams are a little invasive.

SUPER DUPER SECURITY

AN EPIC FAIL.

Bad ResearchWe basically did

Kids will lie for coins

People mis-assessed their English levels

Bandwidth was really a problem

Kids don’t play when their parents are home

“ What happens when you let go of a balloon? ”

PRO TIP #2: HOW TO FIND ACTUAL SPEAKERS OF YOUR LANGUAGE IN ANY COUNTRY:

“ Qu'est-ce qui se passe quand vous lâchez un ballon? ”

DESIGNINGREMOTE STUDIES

Yo Marv, get me some insight on WHY people are bailing out of checkout.

Sure, Boss.

Let’s combine methods. Intercept & observe 10 users live, and do 10 self-moderated with usertesting.com

HEART & SOUL OF REMOTE RESEARCH

Not scheduling participants.

TIME-AWARE TASKS

Told to find purple pumps.

Doesn’t care.

Has to wear this for sister’s wedding this weekend.

Whole different interview.

SCRIPT FOR CUSTOMIZATION

“Please look for a pair of purple pumps in your size.”

“Please look for a pair of shoes you’d like to buy.”

REAL TIME OBSERVERS

WHAT IF IT’S NOT USABILITY?

EXPERIENCE SAMPLING

PROTOTYPES, WIREFRAMES, SKETCHES?

HECK YES.

RECRUITINGFOR REMOTE

RECRUITING OPTIONS

Ethnio via Web or Twitter

Links or house ads to online forms

Recruiting agencies

Customer email lists

Panels

Craigslist/online ads

RECRUITING OPTIONS

Grab someone in the hallway

Grab someone who isn’t directly involved in your project

IM your friends for a quick look

Panels and recruiting agencies

Real customers from marketing lists

Site & offsite intercepts

realness

GOLD STANDARD

Live recruiting

Get as close as you can

USE ETHNIO WHEN

You have Website access

(or Twitter conversations)

Some budget

Quotas are relatively simple

TWITTER RECRUITING

ANATOMY OF A RECRUITING TWEET

@person we’re interviewing [job or characteristic] about [topic] on [dates] (paid research) -interested? [LINK]

I’m looking for [job or characteristic(s)] for a [research activity] on [dates] incentive $X [LINK] please RT

Pro tip #3: edit your profile to mention “UX Researcher” and “Current Study: Martian Life”

USE FORMS WHEN

You can place a link but not code

Low budget

Quotas are relatively complex

Options:Google Docs

Wufoo

Screener constraints

Keep it short

Always ask an open-ended question

Different from surveys

USE AGENCIES WHEN

No website access

People whine about first-time visitors

Higher budget

Online recruiting failed

CUSTOMER LISTS

Sub-segments

Ownership requirements

No website access

PANEL

Would YouUse This?

CRAIGSLIST

When other methods don’t work

No dough

Takes a while

Embarrassing to admit to your friends

LIVE RECRUITING MATH

Target: 6 recruits/hour

Allow for: 1% response (low average

You Need: 10,000 uniques/day

TITLE TEXT

remoteresear.ch/samples“Online Consent”

MODERATED must-haves

Consent

Tech set-up

The beef

De-installation

Thanks & incentive

AUTOMATED must-haves

Expectation setting

Instructions

The beef

Thanks & incentive

Recruiting

Same: Good “talkers”

Same: Demographic fits

Different: Relevant current tasks

Different: Compatible tech

Logistics: What & How

Provide test target

Access

Credentials

Logistics: Who & When

How to connect with the user

Managing time zones

Session length

Paying Incentives

Collect minimal personal info

Deliver quickly

Appropriate for location

STAGED Incentives

For Contextual Inquiry

Start with $75, then at the end offer $25 more for environment photos

Self-made diary studies

EXERCISE #2

Binder PDF Page 6

Use ethn.io or dev.ethn.io (beta) or wufoo

Under 8 Questions

Advanced Bonus: create an automated and moderated screener

okay, draft a screener

Marv-o - what tools should we pick? I mean, tell me what to use for pete’s sake

Sure, boss. There is an AVALANCHE of UX TOOLS.

“At least a little behavioral”

No surveys. No standard web analytics.

CATEGORIES

5

Conceptual

For A Long Time

Self-Moderated

Moderated

Web Analytics

Automated Static

Automated Live

Tasks On Concepts

For A Long Time

People Talk To A Machine

Talk To People

Fancy-Ass Analytics

Tasks On Images

Tasks On A Live Site

I.A. & Navigation

For Ad Agencies

Ammunition

Insight

Just Better Analytics

Still More Ammunition

More Ammunition

Conceptual

For A Long Time

Self-Moderated

Moderated

Web Analytics

Automated Static

Automated Live

Deep Behavioral UX

Insight

Opinions

Conceptual

For A Long Time

Self-Moderated

Moderated

Web Analytics

Automated Static

Automated Live

Mobile

Conceptual

For A Long Time

Self-Moderated

Moderated

Automated Static

Automated Live }

Remote Research ToolkitUNMODERATED (AKA AUTOMATED)

3 parts

Provide test target

Record user behavior

Capture comments (optional)

USABILLA

Clicks & QuestionsHeatmapsYou do your own recruiting$49 - $199

TITLE TEXT

LOOP11 (UN-MODERATED)

TITLE TEXT

GOTOMEETING - $49/MO

TITLE TEXT

WHAT WE USE. A LOT.

• Ethnio + GTM• Usabilla + Usertesting.com• Silverback (Mac) & Camtasia (PC)• Guerrilla mobile

EXERCISE #3Rolling a design for your next study

Page 8 in your binder

Tasks On Concepts

For A Long Time

People Talk To A Machine

Talk To People

Fancy-Ass Analytics

Tasks On Images

Tasks On A Live Site

EXERCISE #4

usertesting.comusabilla - 2 months free (Sign up for an account, but cancel on the payment page. Send an email to [email protected] and mention Nate & Webvisions)

ethnio - 1 month free or trial. CODE: WORKSHOPLOVE

Either “unmoderated” or “practice interviewing”

RESULTS

REMEMBER, BEgraceful

When All Your

TECHNOLOGYBREAKS

WHAT USERS WORRY ABOUT

WHO ARE YOU AND CAN I TRUST YOU?

IF YOU CAN SEE MY SCREEN,

CAN YOU SEE MY FACE?

WILL YOU BE ABLE TO TAKE

CONTROL OF MY COMPUTER?

WILL YOU HELP ME GET THE PLUG-IN

OFF MY COMPUTER AT THE END?

WILL THIS WORK IF I HAVE AFIREWALL/PROXY/STRICT IT

DEPARTMENT?

IS THIS GOING TO END UP ON

YOUTUBE?

So, what could go wrong?

3 points of failure

Technologyyours

participant’s

observer’s

Recruiting

Test Design

Control what you can control

Which is, your set up.

You wouldn’t go backpacking without checking your gear.

Do it 24 hours in advance.

OR ELSE!!

Prepare for whatyou can’t control

User’s dog trips over power cord

Internet affected by local weather

User can’t install

Doesn’t know how to install

Live Recruiting Problems

What if they gave a study and nobody came?

Increase exposure or try social media

When you’re not getting enough views

When the wrong type of users are responding

Increase incentives

When it’s cheaper

When you’re almost getting enough

When segments are tough

Switch methodsWhen base traffic level isn’t high enough

Mid-SessionTech Problems

actual martian?

There’s always a way

If NASA can do THIS from 153 million miles away, we can meet our users where they are, even if we’re a

couple thousand miles away.

[email protected]

@cydharrell

Ethn.io promo code: WORKSHOPLOVE

Thanks for coming.

Go run some awesome research.