Research Operations at Scale (Christian Rohrer at DesignOps Summit 2017)

R E S E A R C H O P E R AT I O N S AT S C A L E : E N A B L I N G O T H E R S W H I L E M A I N TA I N I N G E X C E L L E N C E

C H R I S T I A N P. R O H R E R , P H D

@christianrohrer and crohrer@yahoo.com

https://www.linkedin.com/in/crohrer/

S O …

W H AT C O M E S T O M I N D ?

K N O W N A S A C R E D I T C A R D

C O M PA N Y

B U T I T ’ S A C T U A L LY A B A N K …

G O A L : A T E C H C O M PA N Y

( T H AT J U S T H A P P E N S

T O D O B A N K I N G )

Desirability

ViabilityFeasibility

T O P S T R AT E G Y:

H U M A N - C E N T E R E D D E S I G N

Desirability

ViabilityFeasibility

(AND This means RESEARCH!)

Desirability

ViabilityFeasibility

B U T W H AT A B O U T… E F F I C I E N C Y R I S K R E G U L AT I O N S

D E S I G N R E S E A R C H AT A B A N K I S M I R E D B Y ” G O O ”

Research GooRisk: Strategy Leaks

Risk: Privacy Laws

Risk: Compliance

Reg: File 1099s

Reg: Consumer ProtectionReg: Complaints

Inefficiency: Recruiting & Scheduling

Inefficiency: Technical/Tool

Complexity

AT C A P I TA L O N E , T H E R E A R E D E D I C AT E D R O L E S F O R R E S E A R C H O P E R AT I O N S A N D C U R R I C U L A

Research Operations

• Research Operations Leads: Research request intake, coordinate studies, manage recruitment vendors, scheduling participants, communicating research schedules

• Technicians: manage lab and field equipment, videography

Research Curricula

• Instructors: Develop and teach courses on research methods & procedures to non-research employees

• Specialists: Provide remote research tool licenses, access and training (e.g., survey tools, online studies, diary apps, etc.)

Y O U C A N R U N A S T U D Y L I K E A N

E V E N T

T H I S R E S U LT S I N 3 M A J O R B E N E F I T S

R I S K & R E G U L AT I O N A R E H A N D L E D I N A D VA N C E O R B Y

T R A I N E D P E R S O N N E L

O T H E R “ G O O ” L I K E R E C R U I T I N G , S C H E D U L I N G A N D T E C H N I C A L S E T U P I S

L A R G E LY R E M O V E D

E X A M P L E : “ W I N E A N D D I N E ”

Should you do this?

CONTEXT

R E S E A R C H O P E R AT I O N S M U S T A D D R E S S C H A L L E N G E S F R O M T H R E E K E Y S O U R C E S

Recent Trends in Research

The Context of your business or industry

Universal Challenges inherent in research

What

about

you? IT/INTERNAL

B2BB2C

Major distinctions: • Customers’ needs • Users’ choices • Customer/User Access

Or maybe you’re a startup.

Who are your users today?

What about tomorrow’s users?

Are you getting direct access to your users?

Diffusion of innovation

© 2015 Christian Rohrer

WHAT PEOPLE SAY

WHAT PEOPLE DO

WHY & HOW TO FIX

HOW MANY & HOW MUCH

QUALITATIVE (DIRECT) QUANTITATIVE (INDIRECT)

ATTITUDINAL

BEHAVIORAL

You really should be doing both qualitative and quantitative research

R E S E A R C H O P E R AT I O N S M U S T A D D R E S S T H E K E Y S O U R C E S O F P O T E N T I A L C H A L L E N G E S

Recent Trends in Research

The Context of your business or industry

Universal Challenges inherent in research

Trends in Research

AGILE IS HERE (DEAL WITH IT)• Why? Agile is a must

to attract engineers

• So UCD must fit Agile

• By default, everything goes into a sprint

• Everyone’s on the same team (pod)

• Code early, no docsSource: http://scrumology.com/an-excerpt-from-improving-agility-agile-certifications/

Source:

T O P R E S E A R C H C H A L L E N G E S W I T H T R A D I T I O N A L A G I L E

• No UX roles defined

• Little up front research

• Hard to fit research into a sprint

• Need to feed development

• Dev centricity

R E C O N C I L I N G P R O C E S S E S

Source: Peter

Source: IDEO

Source: Michael James & Luke Walter

Developer & Product Owner Centric

Designer & User CentricNote: all of these are “human-centered” (just for very different humans)

W O R K I N G W I T H A G I L E – 2 M A I N A P P R O A C H E S

• Approach 1: Use agile methodology, but in a separate design phase

• Approach 2: Use an equally attractive, but more user-centered compatible processes (e.g., Lean)

• IMO: Doing strict agile, in which UX is simply added to Dev Pods does NOT work for Human-Centered Design processes. You cannot design or research anything significant in the same sprint as development.

24

E X A M P L E F R O M I N T E L S E C U R I T Y ’ S T R U E K E Y D E S I G N P R O C E S S ( A P P LY I N G A G I L E T O E A R L I E R D E S I G N P H A S E S )

2 weeks CXD sprint2-3 days before sprint starts

What is relevant here:

- Uses the same agile tools and nomenclature

- Design happens BEFORE Development

- Validate/Optimize designs/flows before code (on every sprint)

- Research is scheduled in the ”design sprints” before design executes

Dev

L E A N A P P R O A C H : B U I L D - M E A S U R E - L E A R N

BUILD

MEASURELEARN

Products (MVP)

UsageUptake

L E A N A P P R O A C H : B U I L D - M E A S U R E - L E A R N

BUILD

MEASURELEARN

Products (MVP)

UsageUptake

Improvements

Viable/Desirable

Usability

Interest

Prototypes

Ideas & Concepts

M O D I F I E D L E A N A P P R O A C H : B U I L D - S T U D Y- L E A R N

BUILD

MEASURELEARN

Products (MVP)

UsageUptake

Improvements

Viable/Desirable

Usability

Interest

Prototypes

Ideas & Concepts

STUDY

Source: It’s our Research by Tomer Sharon

Today, researchers can’t be the only ones driving research, or it would be a huge…

Another Trend: Research Participation

D I Y R E S E A R C H

• Over the past 5-6 years, everyone wants to do research:

• All design functions

• Product managers

• Developers

• Even executives!

• Removing barriers to research now essential.

• Just enough training and support are needed

D I Y R E S E A R C H T I P S

• Get the right vendors for you

• Reach the right user tyoe

• Remove goo

• Get tools that work

• Provide training and support

• Incorporate into YOUR process

R E S E A R C H O P E R AT I O N S M U S T A D D R E S S T H E K E Y S O U R C E S O F P O T E N T I A L C H A L L E N G E S

Recent Trends in Research

The Context of your business or industry

Universal Challenges inherent in research

UniversalChallenges

The 5 Purposes of Research1. Understand people and problems

2. Inspire creativity and innovation

3. Validate concepts and solutions

4. Improve and inform designs

5. Assess and measure experience

News flash!These require different skill

sets, not always found

in one person.

T H E TA L E N T P O O L : T H E R E A R E M A N Y S K I L L S A N D TA L E N T S

• It’s hard to find people that can

“do it all”

• Not everybody wants to

• You should allow for BOTH

specialists and hybrid players

on your team

• This enlarges the talent pool

T H E TA L E N T P O O L : S M A L L , G R O W S S L O W LY, V I C I O U S LY F O U G H T O V E R

“We will always be outnumbered.”

-Whitney Quissenberry, panel member at UXPA2016, “UX Past, Present & Future”

• Universities that produce Engineers: 1000s

• Universities that produce UX Professionals: about 75

Conclusion: Don’t waste your design and research talent. Make them as efficient and strategic as you can. Source: indeed.com

W H AT K I N D O F R E S E A R C H W O R K S B E S T F O R D E S I G N E R S ? W H Y ?

W H AT K I N D O F R E S E A R C H A N D I N S I G H T S W O R K B E S T F O R E X E C U T I V E S A N D P R O D U C T P R O F E S S I O N A L S ? W H Y

M A N Y T H I N K S I M P L I S T I C A L LY:

“ B I G N U M B E R S

G O O D . S M A L L N U M B E R S

B A D . ”

“Sorry, but I’ll never make a decision based on research

with just 5 people.” –Product Management

OK, LET’S JUST ACKNOWLEDGE THAT METRICS MATTER

39

• “You can’t manage what you can’t measure”

• “I need a dashboard to control my business”

• “How does our NPS compare?”

• “Invest in data scientists and big data”

• “I want to be more scientific”

• “Build, measure, learn”

• “The design needs to be validated”

Conversion rates, downloads, NPS, customer reviews, A/B testing performance, uptime, abandon rate, analytics data, path/click flows

NOT measuring the experience. They measure the product performance.

A User-Centered Model of User Experience

Look & FeelThe visual and industrial design is clear,

professional and appropriate SoundClear and relevant language, content and information architecture.

The price of entry. In today’s world of technology, data, and design, there is no excuse for

something to be hard.

Ease

Emotional Outcome

It doesn’t matter how easy it is if it doesn’t address the real human need in the right way.

Effectiveness

Sources:

• Rohrer’s Simple Model of UX (2006-2016)

• Forrester: Three Es of CX (2015)40

The PURE Method Pragmatic Usability Rating by Experts

Source: Link goes here

42

PURE: A way to measure “friction”

PURE provides a cognitive load “score” based on : 1. A well-defined user type2. A small set of “fundamental tasks”3. The “performance” of those tasks

(e.g., the Happy Path) for that user type, based on heuristics & design principles

Like in golf, smaller numbers are better, and green is good.

In PURE, we judge the experience based on a ”rubric,” as in skating & gymnastics

• A panel of raters each silently rates an experienceflow they are all witnessing at the same time

• A 1-3 rubric defines howmuch “friction” there is, basedon known UX heuristics

• PURE rates every step of the task (like a “move”)

• Rather thanaveraging scores,the panel decides on the score of each step

Photo credit: Ezra Shaw/Getty Images

A DIFFERENT KIND OF METRIC – ONE FROM A PANEL OF “EXPERTS”CONSIDER AN ANALOGY FROM THE REALM OF MOVIES… THE TOMATOMETER

PURE is kind of like this: An score based on expert analysis

ACTUAL PURE SCORECARD OVER TIME (VERSION OVER VERSION)

45

ACTUAL COMPETITIVE PURE SCORECARD

46

E X E C S W A N T N U M B E R S … S O G I V E T H E M N U M B E R S ! H E R E ’ S W H AT H A P P E N S W I T H P U R E

• People are intrigued. Then, either:

o Owners and Designers of the product want to fix ease of use problems

o Owners and Designers of the product push back, refuting PURE

• Either outcome is good! We are finally focusing on improving an important aspect of user experience: Ease of Use.

• It’s better to have numbers from actual user studies, but if you can’t do it easily, this is acceptable. It will cause positive change in your organization.

K N O W L E D G E M A N A G E M E N T

I S H A R D

R E Q U I R E M E N T S O F A G O O D K N O W L E D G E M A N A G E M E N T S Y S T E M

• Easy to find what we have and know

• Easy to contribute new info and docs

• Must handle more than text docs (shared cognitive artifacts like journey maps)

• Tags and metadata

• Include social and behavioral components (who contributed, who knows more, who used what, who likes what)

• A place where knowledge can grow and be used, not just be stored and accessed

A J O U R N E Y M A P O F K N O W L E D G E

Idea

Question

Research

Finding

Finding

Counter Finding

Counter Finding

Better Idea Another

TestConclusion

Fail

Hypothesis

Pivot

Finding

THEIdea

Winner!

Usability Study

Find & Fix

Find & Fix

Find & Fix

Find & Fix

Find & Fix

Find & Fix

Build &Release

Market data

Success!

T H E P R O B L E M S PA C E I S L A R G E

• Remove risk, deal with regulation, and address other industry-specific challenges

• Teach/empower non-researchers

• Find a way too fit into agile

• Support qualitative research

• Provide quantitative data

• Keep track of and better use your insights across your design process

T H E P U R P O S E O F C A P I TA L O N E R E S E A R C H

We believe opportunity lies in the understanding of the complexity of people.

DEEP EXPERTISE INTERPRETATON EMPOWERMENTPARTNERSHIP

Ultimately, we help bring insights into design.

@christianrohrer and crohrer@yahoo.com

https://www.linkedin.com/in/crohrer/

For more on PURE, contact me or use #puremethod

Thank you