Beyond User Research

Beyond User Research

Lou Rosenfeld • [email protected] World IA Day NYC • 18 February 2017

November 14, 2013: Barack Obama, Commander-in-Chief

November 14, 2013: Barack Obama, Commander-in-Chief

November 14, 2013: Barack Obama, User Researcher-in-Chief

What does victory look like?

User research in today’s organization

Reports from the user research group

Query data gleaned from site search team

XXX.XXX.X.104 - - [10/Jul/2013:10:25:46 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02

XXX.XXX.X.104 - - [10/Jul/2013:10:25:48 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license+plate

Logs from the call center

Reports from analytics applications

Insights from Voice of the Customer research

Reports from CRM applications

Surveys behind Net Promoter Score

Papers from the research center

One agency’s user mental model

Another agency’s brand architecture research

Yet why does so much design still SUCK?

The blind men and the elephant

What Why

Methods employed: quantitative versus qualitative

Goals: help org or users

Organizational goals

Users’ goals

How they use data: measuring world we know versus world we don’t

Measuring the world we knowExploring the world we don’t

...I

Descriptive dataStatistical data

This is true.

The problem with dichotomies

And so is this.

This is true.

The problem with dichotomies

And so is this.

Lou’s TABLE OF OVERGENERALIZED

DICHOTOMIESWeb Analytics User Experience

What they analyze

Users' behaviors (what's happening)

Users' intentions and motives (why those things happen)

What methods they employ

Quantitative methods to determine what's happening

Qualitative methods for explaining why things happen

What they're trying to achieve

Helps the organization meet goals (expressed as KPI)

Helps users achieve goals (expressed as tasks or topics of interest)

How they use data

Measure performance (goal-driven analysis)

Uncover patterns and surprises (emergent analysis)

What kind of data they use

Statistical data ("real" data in large volumes, full of errors)

Descriptive data (in small volumes, generated in lab environment, full of errors)

Four themes for getting to synthesis and insight

1. Balance

2. Cadence

3. Conversation

4. Perspective

1. Balance

XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02

XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license+plate &ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16

Web analytics pro’s question: “Are we converting license plate renewals?”

User researcher’s question: “What are people searching the most?”

Balanced analysis

Thanks

Balance within methods

example courtesy to Angel Brown, Ogilvy DigitalHealth

Thanks

Balance within methods

example courtesy to Angel Brown, Ogilvy DigitalHealth

Rohrer’s user research landscape

Text

Christian Rohrer: http://www.nngroup.com/articles/which-ux-research-methods/

Balance within practice

http://www.nngroup.com/articles/which-ux-research-methods/


Text

2. Cadence

A research cadence from Whitney Quesenbery

Cadence

WeeklyCall center data trend analysis 2 – 4 hours behavioral/quantitative

Task analysis 4 – 6 hours behavioral/quantitative

QuarterlyExploratory analysis of site analytics data 8 – 10 hours behavioral/qualitative

User survey 16 – 24 hours attitudinal/quantitative

AnnuallyNet Promoter Score study 3 – 4 days attitudinal/quantitative

Field study 4 – 5 days behavioral/qualitative

Cadence







Cadence + Balance

Cadence







Cadence + Balance

3. Conversation

Develop a pidgin

Dave Gray’s boundary matrix: http://bit.ly/gWoZQm

KPI

goals

segments

personas

http://bit.ly/gWoZQm

Ban words that impede conversations

• Product names: Omniture,, SharePoint...

• Methods: focus group,, usability test...

• Departments: market research,, analytics...

• Disciplines: business analysis,, information architecture...

• Outcomes: portal, social media layer...

Tell Stories

Tell Stories

SKU: #39072-2AH1

Buy Candy for Strangers

4. Perspective

Maps help us make sense by seeing things in new ways


Text

Avinash Kaushik’s visualization (from Web Analytics 2.0)



“...while I have a bucket for ‘Voice of Customer,’ in hindsight I should have worked harder still to paint the full

qual and quant picture....”

Containers help us make sense by putting things together in new ways

MailChimp’s UX team: drowning in data

MailChimp + Evernote

• Shared bucket of buckets (60 notebooks)

• Email is the API

• OCR’d (nice for SurveyMonkey reports)

• Searchable!

• Led to “regular data nerd lunches”

MailChimp: on the threshold of synthesis

MailChimp + Evernote

• Shared bucket of buckets (60 notebooks)

• Email is the API

• OCR’d (nice for SurveyMonkey reports)

• Searchable!

• Led to “regular data nerd lunches”

MailChimp: on the threshold of synthesis was getting closer to

WeWork’s Polaris: working with a blank slate

Research challenges at WeWork

1. Siloed research

2. Gaps in research memory

3. Reports instead of insight

WeWork: "nuggetization" + metadata


nuggets

metadata



WeWork: filter/search this stuff

•

WeWork: an insight


nuggets

metadata

WeWork’s approach

• Atomic units smaller than reports—nuggets (350 interviews yield @3400 nuggets)

• LOTS of metadata

• Findability improves organizational research memory

• Researcher:curator ratio is 3:1

WeWork’s approach





WeWork’s approach





…is an IA approach

DecisionOps?

DevOps => DecisionOps

DevOps


DevOps DesignOps


DevOps DesignOps ResearchOps


DevOps DesignOps ResearchOps DecisionOps



Q. What will these need to succeed?





A. Information architecture.

How will you get beyond user research?


How to move your organization from micro-decisions to major insights?



Who needs to be talking with whom?




Can balance, cadence, conversation, and perspective help?





What kind of operations will get you there?






How will information architecture help?






How will information architecture help?

Thanks!

slides: http://rfld.me/11FrI3o

article: http://rfld.me/145ZccP

Lou Rosenfeld

@louisrosenfeld

www.rosenfeldmedia.com • @rosenfeldmedia

http://rfld.me/11FrI3o

http://rfld.me/145ZccP

http://www.rosenfeldmedia.com