Mixed Methods Research in the Age of Big Data

A Primer for UX Professionals

Zachary Sam ZaissUX Data Scientist | Microsoft Cloud

@zszaiss

2006 2012 2016UX Researcher UX DS

Berkeley MIDS

Gartner Hype Cycle for Emerging Technologies: 2014

http://www.gartner.com/newsroom/id/2819918

Gartner Hype Cycle for Emerging Technologies: 2015

http://www.gartner.com/newsroom/id/3114217

The Education Perspective

https://whatsthebigdata.com/2012/08/09/graduate-programs-in-big-data-and-data-science/ http://uxmastery.com/resources/ux-degrees/

84 78Graduate Degree

Programs inData Science

Graduate DegreePrograms in UX

http://radar.oreilly.com/2013/10/design-thinking-and-data-science.html

We need to makecollaboration with

Data Scientists a priority…

… and it starts with aconversation

Tip #1

Stake Your Claim

Qualitative Evaluation Criteria Talking Points

Quantitative basis for n values

https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ http://www.measuringu.com/blog/five-history.php http://www.measuringu.com/blog/five-for-five.php

Qualitative Evaluation Criteria Talking Points

Quantitative basis for n values

Existence Proof

https://www.youtube.com/watch?v=3uqZPnxG4_w

Qualitative Evaluation Criteria Talking Points

Quantitative basis for n values

Existence Proof

Grounded TheoryInductive vs. Deductive Reasoning

http://www.slideshare.net/traincroft/hcic-muller-guha-davis-geyer-shami-2015-0629

Theory from Data

Data from Theory

Qualitative Evaluation Criteria Talking Points

Quantitative basis for n values

Existence Proof

Grounded TheoryInductive vs. Deductive Reasoning

Constructivism vs. Determinism

https://us.sagepub.com/en-us/nam/research-design/book237357

Discussing evaluation criteriafor qualitative research

needs to be second nature.

Example

What is yourmic drop moment?

Tip #1

Stake Your Claim

Tip #2

Speak the Language

vs

Models + Key Aspects of Analysis

Descriptive ModelDescr ipt ive Stat i s t i cs

Statistical SignificanceWhat is the probability of obtaining this result given the null hypothesis is true?

Practical SignificanceIs the effect on the outcome large enough to be considered relevant?

http://fivethirtyeight.com/features/statisticians-found-one-thing-they-can-agree-on-its-time-to-stop-misusing-p-values/

The statement process was lengthier and more controversial than anticipated.

6 Principles for p-values from ASA’s Statement

1. P-values can indicate how incompatible the data are with a specified statistical model.

2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.

3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value crosses a specific threshold.

4. Proper inference requires full reporting and transparency.

5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.

6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108

Models + Key Aspects of Analysis

Descriptive ModelDescr ipt ive Stat i s t i cs

Statistical SignificanceWhat is the probability of obtaining this result given the null hypothesis is true?

Practical SignificanceIs the effect on the outcome large enough to be considered relevant?

Predictive ModelSuperv i sed Machine Learn ing

AccuracyHow well does the model predict the outcome for new data cases?

https://www.captionbot.ai/

Models + Key Aspects of Analysis

Descriptive ModelDescr ipt ive Stat i s t i cs

Statistical SignificanceWhat is the probability of obtaining this result given the null hypothesis is true?

Practical SignificanceIs the effect on the outcome large enough to be considered relevant?

Predictive ModelSuperv i sed Machine Learn ing

AccuracyHow well does the model predict the outcome for new data cases?

Representation ModelUnsuperv ised Machine Learn ing

Optimization CriteriaHow will we determine that we’ve built a reasonable and appropriate representation model for our data?

vs

None of these measuresget at the contextual meaning

behind the model.

A Diagram for Product Manager…

Source: Martin Eriksson, Mind the Product. http://www.mindtheproduct.com/2011/10/what-exactly-is-a-product-manager/

… And a Framework for Attributes

UX

Business

TechExperience Attributes

Customer attributes that can explain how

that customer will experience a product.

Technology AttributesCustomer attributes that can explain whether customers will have technical issues with a product.

Business AttributesCustomer attributes that can explain the extent to which the customer will contribute to business outcomes.

Example: Developer Tools

X

B

T

Prog LanguageTarget PlatformProject Complexity

Project AudienceType of App

Educational Background

Keyboard ProclivityProject Complexity

Example: Freemium Games

X

B

T

Platform UsedFacebook Connected

Whale Status

Completionist Tendencies

Game Session Time

Example: Fitness Bands

X

B

T

Connected DevicesType / Version

Frequency of Exercise

Friends with Same Band

Finger Shape (Fat Fingers)

FarsightednessSkin Irritation

We are uniquely qualifiedto articulate the experienceattributes of our products.

Tip #2

Speak the Language

Tip #3

Get Involved

A Metaphor for A/B Experiments

A Better Metaphor for A/B Experiments

XX

X

XX

X

X

How can we providegreater context toA/B test findings?

Heterogeneous Treatment Effects

control treatment

some kpi

0.71

0.72

productexperts

productnovices

control

treatment

converted didn‘t convert

converted didn‘t convert

control

treatment

converted didn‘t convert

converted didn‘t convert

Heterogeneous Treatment Effect

https://datadialogs.ischool.berkeley.edu/2014/schedule/experiments-action

Tip #1: Stake Your Claim

Tip #2: Speak the Language

Tip #3: Get Involved

Mixed Methods Research in the Age of Big Data

A Primer for UX Professionals

http://www.uxpa.org/sessionsurvey?sessionid=113