Research for Different Levels of Design · Source: Peter Merholz and Kristin Skinner’s “Org Design for Design Orgs” ... Design or redesign a website or mobile app Strategy,

CX NYC 2018

Kelly Price, Analyst

Research for Different Levels of Design

2© 2018 Forrester Research, Inc. Reproduction Prohibited

Agenda

• The role of research in CX

• How to get the insights you need to succeed

• How to adjust your research plan

• Discussion and Q&A


Mastering six competencies leads to great CX


But getting to great starts with research


What do we mean when we say

“research”?


Source: Chart adapted from Tricia Wang :https://medium.com/ethnography-matters/why-big-data-needs-thick-data-b4b3e75e3d7

Data-driven

Thick data

Depth of insight

N

Insights derived from

quantitative methods using

large samples

Analytics, ML,

Big Data…



Data-driven

Thick data

Depth of insight

N

Human-insights

led



large samples

Analytics, ML,

Big Data…



Data-driven

Human-insights

ledThick data

Depth of insight

N

Deep understanding derived from

qualitative research conducted with

smaller sample sizes



large samples

Analytics, ML,

Big Data…

9© 2018 FORRESTER. REPRODUCTION PROHIBITED.

“Good inventors and designers deeply

understand their customer. They spend

tremendous energy developing that

intuition. They study and understand many

anecdotes rather than relying only on the

averages you'll find in surveys.”

- Jeff Bezos’ 2017 Letter to Amazon Shareholders


Where does this actually come

into play?


Typically people start with methods…

Website redesign


…but this is jumping the gun

Website redesign

What problem

am I solving?


Design happens at different levels of scale

Source: Peter Merholz and Kristin Skinner’s “Org Design for Design Orgs”


Asking the right questions starts with identifying the level of design

Example Challenges Level

Identify what opportunities exist to improve our

customers’ lives – what should we be designing?

Big picture

Reimagine a customer journey (e.g. onboarding) Strategy, Structure,

Surface

Design or redesign a website or mobile app Strategy, Structure,

Surface

Evaluate whether our site navigation matches how

users model content

Structure

Align our website design with our company’s new

brand guidelines

Structure, Surface


Research basically answers four core questions

What problems or

opportunities

exist?

What’s the best

way to solve this

problem?

Are our solutions

correct?

What’s happening

with this

experience?


Challenges map to specific questions

What problems or

opportunities

exist?

What’s the best

way to solve this

problem?

Big Picture Strategy Structure | Surface

Are our solutions

correct?

What’s happening

with this

experience?


Challenges map to specific questions

What problems or

opportunities

exist?

What’s the best

way to solve this

problem?

Big Picture Strategy Structure | Surface

UNCERTAINTY/PATTERNS/INSIGHTS CLARITY/FOCUS

Are our solutions

correct?

What’s happening

with this

experience?


Once we know where we’re starting we can get on the path to the right insights

Design Challenge Core research question

Research goalsResearch questions

Methods Informed design decisions


What problems or

opportunities exist?What’s the best way to

solve this problem?

Are our solutions

correct?

What’s happening

with this experience?

Example: Research on Retirement SavingsCore

question

related to

design

challenge:


What problems or


solve this problem?

Are our solutions

correct?

What’s happening


Research

goal:

Understand

customers’

retirement goals

Identify behaviors that

prevent customers from

achieving their goals

Example: Research on Retirement Savings

Understand what features

increase customers’

retirement contributions

Get insight into whether

customers are meeting

their goals

Core

question

related to

design

challenge:


What problems or


solve this problem?

Are our solutions

correct?

What’s happening


Core

question

related to

design

challenge:

Research

goal:

Understand

customers’

retirement goals










their goals

Specific

question(s):What does “retirement”

mean to our customers?

Are people able to find

the information they

need?

How are my

experience KPIs

tracking over time?

What information are people

looking for when they interact

with us about retirement?


What problems or


solve this problem?

Are our solutions

correct?

What’s happening


Core

question

related to

design

challenge:

Research

goal:

Understand

customers’

retirement goals










their goals

Specific

question(s):What does “retirement”

mean to our customers?

What information are people

looking for when they interact

with us about retirement?

Are people able to find

the information they

need?

How are my

experience KPIs

tracking over time?

Methods: Discovery Definition Evaluation Measurement


Discovery Definition Evaluation Measurement

Exa

mp

le M

eth

od

s

Field observation/ethnography

Diary study

Contextual inquiry

Cultural probe

Experience sampling

Cognitive

mapping

Card sorting/tree testing

Participatory design/co-creation

Concept mapping

Usability testing

Surveys

Competitive analysis

Interviews

Benchmarking

WoZ testing

Cognitive walkthroughs

A/B/ multivariate testing

Analytics data

KPI tracking

Search log review /FAQ review

Support call logs

Eye-tracking

VoC feedback

Usability bug review

Heuristic evaluation

Existing research synthesis

Biometrics (EMG, GSR)

What problems or

opportunities exist?What’s the best way to solve

this problem?Are our solutions correct? What’s happening with this

experience?


What happens when we jump

ahead?



The Result?

$1.85BEstimated $1.85 billion

losses in same store sales


Common barriers to “ideal” research

$ Budget

Time

Buy-in

Regulations

Access to

participants


Example: Overcoming Challenges

Design challenge:

Improve the journey of researching and

applying for a checking account, to drive sales

What the research team wanted to do:

Diary study

Challenges:

3 weeks to do research,

< $3K budget,

compliance not comfortable with diary tools

What research did instead:

• Mapped using existing data

and insights, focused

research on filling gaps

• Remote ‘retrospective’

interviews over Web Ex with

target customers

• Total cost: < $2500


Example: Overcoming Challenges

Design challenge:

Improve the journey of researching and

applying for a checking account, to drive sales

What the research team wanted to do:

Diary study

Challenges:

3 weeks to do research,

< $3K budget,

compliance not comfortable with diary tools

What research did instead:

• Mapped using existing data

and insights, focused

research on filling gaps

• Remote ‘retrospective’

interviews over Web Ex with

target customers

• Total cost: < $2500

Key: Be willing to get “scrappy”, but do

so within the appropriate research

“swim lane”

STAKEHOLDERS

STUDY BACKGROUND RESEARCH QUESTIONS METHODS

PARTICIPANTS

Business Leaders• Who’s buy-in are you going to need to bring this

research plan to life?

• How will you frame the value and importance of

this research to get them on board?

• How will you involve them in the research process

to continue to prove the value?

Subject Matter Experts• Who has subject matter expertise for the

questions you are answering?

• What information might they have to help you form

stronger hypotheses?

• Are there places you can pull them into the

research process to help design better questions,

find better participants etc.?

Facilitators• Who will help plan, organize, and execute the

research?

• Who can help prove value creation or help link

research to success metrics?

Implementers• Who are the decision makers who will use the

research to make changes to the experience?

• How will you involve them in the process to

increase their buy-in?

• What is the context of this research? Why is it being done? • What are the 5-10 questions you want to have answered by the end of this

project?

• Why are these questions important?

• Do you have any existing research or data that provides insight into any of

these questions?

Identification• What methods will be used?

• How will each method apply to the research questions

you’ve identified?

Planning• What methods will you manage internally? Will you work

with any outside research providers?

• Where will the research take place? Will you need to

arrange transportation, or book space?

• How much budget do you have to allocate, and how will

that be divided across the different methods employed?Attributes • What attributes will you use to identify and recruit? Why are these attributes

important?

Recruitment• How will you locate and schedule participants? Will you work with an outside

provider?

Incentives• Will you offer incentives for participation? If so, how much? How will the

incentives be delivered?

Created by: Version:Date:

Source: Forrester Research, Inc. Unauthorized reproduction, citation, or distribution prohibited.

Project Tittle:

• Why are we conducting research?

• What are the 2-4 goals we have for this project? Why are

these goals important/justified?

RESEARCH GOALS

SCHEDULE

• What is the timeline for this project?

• What milestones will you meet along the way, and at what

dates? (e.g. recruitment, running methods, analysis)

SUCCESS METRICS

• What behavioral indicators will you use to provide

evidence that research findings are being used?

• What metrics will you use to provide evidence that

research lead to tangible experience improvements?


Discussion & Q&A

Thank youKelly Price

[email protected]


Appendix


How do we get from design challenge to design answer?

We start with a

challenge or a

problem



We have a goal we

want to achieve



Solving the problem

and achieving our goal

requires the right

human insights


Questions guide our research approach

What is the right design? Is this design right?

Discovery Definition Evaluation Measurement

Documents

Research for Different Levels of Design · Source: Peter Merholz and Kristin Skinner’s “Org Design for Design Orgs” ... Design or redesign a website or mobile app Strategy,