Webinar Mobile Research: Are the results valid?”

Gerard Loosschilder

CIO & Partner

Robin de Rooij

Senior Manager

Abigail Joffre

Today’s webinar host

#SKIMwebinar

Share your thoughts online:

Mobile Choice-based Conjoint

A valid alternative

Mobile surveys: it’s time to act

Increasingly people take surveys on a

mobile platform. Filtering them out

may skew the sample.

2

Source: Burke, July 2013

89%

4% 7%

2013

Smartphone

Tablet

PC

Screen

In

Out

Biased sample

Standard

survey design

Mobile survey

design

% of people taking

the test survey by

platform

SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

So we better format right…

3 SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

Introducing 3*3 mCBC

Today’s presentation is about the

validity of mobile choice-based conjoint

4 SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

Adapting to the shorter attention span of the mobile user

By reducing the number of choice tasks and number of choices per task

5

Three choice tasks Completed in

Three options each

SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

At the same valid results as a traditional CBC exercise

The good news

The MAE of share predictions is

<4% at the correct sample size

Overall results, conclusions and

recommendations are the same

The bad news

So far, it is impossible to reduce

the number of choice tasks and

options per task to fewer than the

3*3 platform

6 SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

3*3 mCBC requires a larger sample size

As a rule of thumb, sample size is

tied to number of parameters to be

estimated, determined by the

number of attributes and levels

Traditionally, we recommend a minimal

sample size of n= 200 for any conjoint

study. Now, the sample size goes up

with the number of parameters to be

estimated.

If # of

parameters is

Sample size

n=

12 (3 atts*4 levels) 400

16 (4 atts*4 levels) 800

20 (5 atts*4 levels) 1,200

25 (5 atts*5 levels) 1,600

7 SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

3*3 mCBC requires a well-balanced research design

In a 3*3 mCBC research design,

the number of data points

collected is much smaller than the

number of parameters to be

estimated.

So, the key success factor is a

high-quality research design

Frequency balanced

Every combination of two

attributes and their levels is

represented well by all research

designs

Market knowledge included

We take current market shares

and knowledge about consumer

preferences into account in design

generation

8 SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

How do we know if mCBC

using a 3*3 design is valid?

Empirical evidence

9 SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

Because we have validated it …

We determined the relation between

sample size and convergent validity

It shows that at increasing sample sizes,

mobile CBC is a good approximation of

common CBC on traditional platforms

10 SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

… in a study into pain killers and batteries

11

What is the validity of mobile CBC at various sample sizes

if compared with the same conjoint study in a traditional way at n= 200?

Attributes:

Brand

Price

Size

3-5 levels each;

12 parameters

Attributes:

Brand

Admin method

Dosage

Package

Price

3-5 levels each;

16 parameters

SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

Attribute importance values show great similarities at all sample sizes for mobile CBC

12

The values for

batteries are

equally similar

Variations in

attribute

importance values

are insignificant

0%

10%

20%

30%

40%

Att

rib

ute

im

po

rtan

ce

Platform and sample size

Pain killers

Brand

Admin method

Dosage

Package

Price

SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

However, at larger sample sizes, preference shares get closer to benchmark leg, demonstrating greater validity

13

Mean Absolute Error

(MAE) is a metric

expressing how far

apart the preferences

shares are as

measured by mCBC at

increasing sample

sizes compared with

the benchmark CBC

at n= 200 0.0%

2.5%

5.0%

7.5%

10.0%

200 300 400 500 600 700 800 900

MA

E v

alu

e i

n %

Sample size in the mobile leg

Mean Absolute Error values (MAE, in %)

Batteries

Pain killers

The break point is at n= 400; twice the

sample size of the benchmark

Especially at pain killers at 16

parameters (instead of 12 for batteries)

SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

Conditions to make 3*3 mCBC work – research design

Every display of product profiles in a choice task is valuable. It

should not be wasted on prohibitions or alternative specific designs.

• We can not tolerate any reduction in D-efficiency values

So for now, we can apply mCBC in some standard situations.

We do not apply 3*3 mCBC in situations in which there are

prohibitions, utility balanced designs and alternative-specific

designs

14 SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

Conditions to make 3*3 mCBC work – real estate

15

There is no space for large

amounts of information

Max. 5 attributes

Text is succinct, keywords

only

No complex descriptions

Pictures are

icons or schematic

Be careful with package or

ad testing

SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

16

Interaction design

http://tinyurl.com/skimmcbc

Powered by

SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

3*3 mCBC in Price Sensitivity Analyses – batteries

17

http://tinyurl.com/skimmcbc

SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

3*3 mCBC in Price Sensitivity Analyses – pain killers

18

http://tinyurl.com/skimmcbc

SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

3*3 mCBC in Telecom – design of a postpaid portfolio

19

http://tinyurl.com/skimmcbc

SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

Mobile in MaxDiff applied to Semantics and other tests of claims and benefit statements

20

http://tinyurl.com/skimmcbc

SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

We capture actual consumer behavior in a mobile shopping exercise taken on mobile devices

Packaging can be seen in detail through

a zoom-in feature

The shopping basket selections can be reviewed

SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative” 21

The exercise allows us to optimize features of the mobile retail environment and its products

22

Product rankings in

search results

Ratings & Reviews

Price & Promotions

Images and

iconography

Product

descriptions

and sizes

24 hour

protection Product callouts

Shipping options

SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

Replicating online retail environments transcends to services offered by telecom networks

23

A build-your-own post-paid subscription environment inspired by Hi,

a subsidiary brand of KPN in NL

A build-your-own post-paid subscription environment inspired by Tele2,

an independent price fighter brand in NL

SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

… and by publishing houses…

24

The original website of the publishing house to have consumers sign up for a subscription …

… converted into a conjoint choice exercise to mimic the online environment

SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

3*3 mCBC is a valid addition

To conclude

25 SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

To conclude – mCBC is a valid addition

Do

• Take market knowledge into

account

• Use a larger sample

• Use succinct text, iconized

visuals

• Have five attributes at five

levels max

Don’t

• Use it for large CBCs

• Include research design

restrictions

• Use it for package, ad or

concept work

26 SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

Next steps

• SKIM researchers will continue to validate mCBC …

• For allocation studies

• For large research designs using consideration sets

• For more complex visuals

• Keep improving UI for Mobile CBCs

• Across dominant platforms

• Translate learnings back to any CBC

27 SKIM Webinar “Mobile Choice-based Conjoint: A valid alternative”

Go to www.skimgroup.com/webinars

for today’s presentation slides and more!

Robin de Rooij Senior Manager

r.derooij@skimgroup.com

@Robin_de_Rooij

Abigail Joffre a.joffre@skimgroup.com

+1 201 963 8430

@SKIMgroup

Gerard Loosschilder Chief Innovation Officer

g.loosschilder@skimgroup.com

@gloosschilder

#SKIMwebinar

Share your thoughts online: