Online Research Coming of age - Brownbag Presentation at Universitty of Pretoria SA

Holger Lütters - Online Research Coming of Age - University of Pretoria September 2014

Prof. Dr. Holger LüttersHochschule für Technik und Wirtschaft Berlin, Germany

Online Research Coming of AgeBrown-Bag presentation 18.09.2014


Holger Lütters – a little older than the internet

My unplanned way into online research • First E-Mail sent 1994• First Webpage 1996• 1999 Own online survey created with

HTML • 2004 PhD on web based Analytic

Hierarchy Process • 2006 Interactive SniperScale• 2007 Panel study• 2008 International Multi-language

studies• 2010 Facebook for recruiting• 2011 Mobile only study• 2012 Avatar study • 2013 Gyroscale use • 2014 Mobile only Panel


Problem with online market research:Lack of representativeness

Quelle: Fessel GfK Austria


Online Market Africa: SA at 50% of Internet penetration leading on the continent, but still away from a representative popuation

http://www.internetworldstats.com/stats1.htm


Methods of online research are not a substitute but a complement to the existing methods of market research

• Online market research has established itself as a serious approach (already 1st place in many countries)

• There are outstanding online solutions for many questions available

• For some topics online still is an inappropriate method due to the lack of representativeness of the population

• The time continues to work for the online market research idea:

• Expiration of classical structures of accessibility of people on the phone (5 phones in the home, multiple phone numbers, only mobile users, VoIP, etc.)

• Rising costs for traditional types of surveys

• Many online groups are already covered at 100% (students, teenagers, business surveys)

• Everything presented today will be in South Africa sooner or later. With our university cooperation it may be sooner and give you a competitive advantage over other institutions in your country and will help you compete with the international research community


Fundamentals of online research

1


„Internet as an object“

Analysis of online users, online sites and online research

methods

„Internet as a method“

Collection of market and customer data with the help of

the Internet

Online Research: Internet as an object or a method?

Source: Zerr

Online-Research

Non-reactive data• User-Tracking• Visitor Analysis• Conversion Analysis• …

Reactive Data• Interviews• Focus Groups• Web-

Experiments• Delphi• …


Online Market Research: Primary and secondary research with the help of Internet technologies

Example- Reports- Statistics- Databasesin - WWW- Intranet- Extranet- Social Media

ReactiveData

Non-reactiveData

Primary researchInternet as a tool

Secondary research (Desk)Internet as a source

Online-Market Research

Social Media Research


Online-interview = location-independent, impersonal, self-administered computer-interview

Face-to-Face Telephone

impersonalpersonal

Computer-based surveys

Computer AssistedPersonal Interview

CAPI

Computer AssistedTelephone Interview

CATI

Computer AssistedWeb-Interview

CAWI

Many methodologically supported interviews are computer-assisted interviews ...... with the respective advantages and disadvantages of the specific interaction method


5 standard survey modes in 2014

Advantages of the different survey methods?Every method has its own purpose and target group

Effect of the chosen interview mode on results?Effect on cost, time, quality?

Personal Interview

Paper PencilSurvey

Telephone

Interview

Online

Interview

Mobile Interview


Three steps to run a surveyonline

2


Three key challenges arise after the decision for an online market research project

(Scientific) Modeling of

questionnaires

Development of a survey design

Programming

Distribution & Response

Management

Reach the defined

participants

ControllingScreen-out Quota Definition

Analysis & Reporting

Statistical analysis

Communication of results


Evaluation of an online software for your own research project

Make sure that the software…

• Does what you need

• Has certain methodical features (Conjoint; PSM; AHP)

• Can be connected to a panel / CRM-System / Facebook

• Gives you total control over the entire study process (example: no advertising from third parties; no access from outside; no logos etc.)

• Creates appealing questionnaires

• Is accessible during and a certain time after the actual study (at least for more than 4 weeks / for free)

• Is exporting the data to SPSS/Excel

• Is easy to use (for surveymaker AND surveytaker)


Online survey software should be online (SaaS)Some providers offer free access for non-commercial scientific research

http://www.gesis.org/dienstleistungen/methoden/beratungen/datenerhebung/online-umfragen/software-fuer-online-befragungen/#listings

Freeware:


Creatingmotivatingquestionnaires

3


Don’t worry about the duration. The first minute is the most important!


How do users perceive the interview experience?Leaving HTML-Standards towards bigger elements grows usability

HTML

AJAX


Why taking over paper pencil standards onto the color screen?New forms are available and show very positive effects

Simple Radio Button

Simple Slider Design

Slider Design with Graphical Support

Flash Scale

Lütters, H. / Westphal, D. / Heublein, F. (2007): SniperScale: Graphical scaling in data collection and its effect on the response behavior of

participants in online studies; General Online Research Conference (GOR) 2007, Leipzig.


With growing bandwith more appealing questionnaires can be createdEffect: Higher willingness to participate


Lütters/Westphal/Heublein (2007): “The future of graphical scaling is interactive”

Flash interaction with movement and sound

Non-interactive scales with AJAX technology

Interactive tests of the team based on Flash technology in 2007 Examples based on Ajax technology


Audience adopted design creates higher response rates


Adopt to the audience and the device used: Example of first iPad-only study


The progress bar: Never ending story of misunderstandings

Summary: • Has a strong effect (any kind of

undesired effect is negative)• Makes people thing about the

duration more than they should• Is only motivating when it is faked

(53% after 2 questions)• Faking vs. AAPOR/ESOMAR code of

ethics• People believe that it is a necessary

part of a questionnaire. It’s not!• Avoid the progress bar!

Try the count-down…

http://conftool.gor.de/conftool11/index.php?page=downloadPaper&filename=Callegaro-hould_we_use_the_progress_bar_in_online_surveys_A_meta-analysis-151.pdf&form_id=151&form_index=&form_version=final&CTSID_GOR11=AgPbzZ3ygg6WVMGwtHF8Y0


Mobile research about to break through.Device Agnostic Research is the way to go on the long term


The questionnaire designed often looks different to interviewee than expected by the designer

Source: pangea labs; Dillman 2000: 360

OnlineQuestionn

aireDesigner

Designer‘s Hardware

& Software

Questionnaire

as seen by Designer

Questionnaire

as seen by Designer

Respondent‘s

Hardware &

Software

Online

Respondent

Questionnaire

as seen by

Respondent

Questionnaire

as seen by

Respondent


Inviting people to online questionnaires

4


Where will the participants be coming from?Why should they participate in such a survey?

• Employees• Customers• Company Website• Other Website• Convenient Sample

(Ponzi scheme )• Riversampling• Panelrecruiting• Offline•…

1. Topic2. Incentives

(monetary / non-monetary)3. Offering Results4. Design5. Anonymity6. …

• Open vs.. closed• Quota features• Plausibility check• Consistency Check• …

Controlling

of sources and quality of answers

Motivation

(to fully complete)

Sources of respondents


Motivation to complete starts even with the invitation approachInfluence on opening rate and click rate

Sarah? Isabel?


SPAM as a central problem of email invitationsHTML vs. Plain Text:


Social Media Recruiting is an opportunity which needs some extra care

0

20

40

60

80

100

120

140

160

An

za

hl

de

r S

tart

er/

Co

mp

lete

res

po

ns

es

Completes Social Media

Starter Social Media

14.2. 23.8. 5.10.

Daily average completion rate:

80%

First reaction to blogpost on web

First Invitation

as post on

adidas Running

Reminder

(Second Invitation)


Combining offline and online with incentivesThe GAP invites to the customer survey on the receipt.

http://outlet-feedback.gap.com/


Controlling of online interactions

5


Controlling is important to see success and understand effects

datetotal no ofcontacts

quota full(interview terminated by questfox)

screened out(interview terminated by questfox)

No ofinterviewspossible Completed n= Completed %

26.08.2013 41 0 1 40 40 100 %

27.08.2013 175 0 3 172 159 92,4 %

28.08.2013 235 103 1 131 103 78,6 %

29.08.2013 1041 639 14 388 166 42,8 %

30.08.2013 1280 976 26 278 194 69,8 %


Quality is insured by monitoring


Controlling each question tells about problems of the questionnaire

question pageresponsesto question page

surveycompleted

Q1: ### Steuerung der Befragung 2772 (99,9% of total) 662 (23,9%)

Q2: Screener (Geschlecht) 2726 (98,3% of total) 662 (24,3%)

Q3: Screener (Alter) 2613 (94,2% of total) 662 (25,3%)

Q4: Screener (Bildung) 2597 (93,6% of total) 662 (25,5%)

Q5: Screener (Bundesland graphisch) 2576 (92,9% of total) 662 (25,7%)

Q6: Screener (Stadt oder Land) 2557 (92,2% of total) 662 (25,9%)

Q7: Screener (Location) 2547 (91,8% of total) 662 (26%)

Q8: Erfahrung mit Befragungen der Marktforschung 2536 (91,4% of total) 662 (26,1%)

Q9: Screener (Teilnahmebereitschaft) 2528 (91,1% of total) 662 (26,2%)

Q10: Screener (Device) 2773 (100% of total) 662 (23,9%)

T11: Politik, Liebe, Drogen 719 (25,9% of total) 639 (88,9%)

Q12: ### P01 - Regierung weiterempfehlen 707 (25,5% of total) 641 (90,7%)

Q13: P02 - Wo stehen Sie politisch? 700 (25,2% of total) 644 (92%)

Q14: P02_Sonntagsfrage 695 (25,1% of total) 642 (92,4%)

Q15: ### S01 - Beziehungsstatus 689 (24,8% of total) 641 (93%)


Quota control attempts to portray the planned number of representative cases and close the study in case of reached limits

Example quota monitoring during an ongoing study

Example quota achievement after completing a study


IP to location creates detailed reports for regions (without asking)

Result of a UK study recruited over a panel


Buying the respondents‘ willingness to participate

6


Example of a worst case scenario: Recruiting from company database2780 email invitations resulted in 27 completed interviews

PERSONAL INVITATION INFO @ INVITATION

Source: B2B research project in EU research

Response to an international survey (6 languages) after one week

Σ=27


Recruitment: Make or Buy?Recruiting for the same study with a commercial panel

Σ=638

Connection to a commercial panel in the same period

638 full interviews from panel vs. 27 full interviews with company e-mail-addresses

Source: B2B research project in EU research


Many service providers are established in the new market of recruitment of participants (so-called online panel or MROC)


The price of ONE participant in an online study is a calculation based on duration and incidence of the targeted criteria

Example from SmartResearch in Euro: 1 € = 14 SAR

Number of questions

Panel

Number of Interviews

Incidence

9500 SAR

Drivers of the price• Duration (calculate € 0,10 cent per

Minute) thus less than 15 rand for a complete 10 minutes interview

• Only complete interviews are paid• 2 Minutes sampling time allowed to

define screen-outs and quota rulesIncidence:• 50% Male / Female• Car Owner: 70%• Owner of a Volkswagen: 10%

(equal to market share)• Owner of a VW Polo 1%In words: you need to contact 100 people to get one polo driver into your study. And what if he/she drops out?


The willingness to participate is guaranteed through incentivesThis approach creates less topic involvement


Future of sampling: AYTM All-in-One-Solutions per Click


Google consumer surveys is a combination of survey making and sampling at the same time


panelfinder.com as a meta platform for all kinds of panels


Set up minimum standards and monitor your panel.Good Panels have full control over the relation to the participants


Summary: Panels are very functional and cause less organizational pain… but need to be paid

Hints:• Always ask for 3 quotes from different companies and ask for their

calculation of the INCIDENCE• Only negotiate on full interview• Start small (n=300) -> you can still buy more respondents later• A panel telling you “no problem” might not be your first choice

Suggestion: • Why not doing general research in the USA (cheapest) or Europe (widely

available)?• Why not building up the first university panel in South Africa?


A few tips for your online project: Serious fun®

• Select the online method only when it is appropriate (not for financial reasons)• Try on-line studies: you will learn a lot about yourself and your relationship with

employees and customers or other groups much quicker• Take your time: An online project requires a much time as a classic data

collection project. Perhaps a little more when doing it for the first time• Emphasize more on methodology than on technology• Take care of your population. They are interviewed more often than

needed. Reconsider the number of required interviews in every study• Take care of every single respondent. Rethink difficulty, design, duration for

each project. Find an average between fun and torture.• Think from the perspective of your participants: Try to take into account an

external point of view throughout your project. Is it really as easy as you think?• Remain critical: Do not just mindlessly accept approaches from the past or

from offline into this new world. Go into a critical review process after a project • Online is different. Online needs to be different! Take advantage of the

benefits of a new world of data collection opportunities• Be open to new ideas. The internet is faster than you are. Try to incorporate

some of the brilliant ideas into your research


Newer forms of online research

used by HTW Berlin Marketing Research Team


Eyetracking: Registering peoples looks

Eyetracking of a single person Heat Map as aggregation of views


Face Detection technology from the Fraunhofer InstituteGfK EMO Scan is the commericalised version

Idea: Facial Scan Field of application

• Gender recognition• Estimate of age• Mood measurement• Recognition of people in

groups• Recording the changes in

mood on exposition to changing stimulus (ex. Video ad)

• Possible with simple technology

http://www.youtube.com/watch?v=Fs-ngMK09Vk

http://www.youtube.com/watch?v=Fs-ngMK09Vk


Listening is the new asking (Poynter)Insights from social media without asking anyone anymore


Interaction through device movement: The Gyroscale literally moves answers

The Gyroscale® is available in questfox®Source: www.pangealabs.com Gyroscale® and questfox® are trademarks of pangea labs GmbH Switzerland

http://www.pangealabs.com/


The avatars are empowered by text-to-speech technology which allows them to read out loud written texts

Different people have very different relations to ....

Text for reading:

Text is read by Avatar:

R1

R2

R3


Outlook into the future ofonline research


Some trends from the US are online on ist way to Europe

http://issuu.com/researchshare/docs/grit-winter-2014


#newMR is discussing the future of online

http://www.mrweb.com/mrt/

http://www.mrweb.com/mrt/


Greenbook as a good online source to start with

http://www.greenbook.org/


What happens to market research, if the boundaries are broadened?

“In the future many professions and sectors will go the way of steam engine makers and the man who used to walk in front of cars carrying a red flag.

This might include the definition of market research as a business sector.”

Discussion under http://krz.ch/vkib

http://krz.ch/vkib


Thank you for your interest Feel free to contact me

Holger Lütters – HTW Berlin

Contact informationProf. Dr. Holger LüttersHochschule für Technik und Wirtschaft Berlin

[email protected] Twitter @luettersxing.com/profile/Holger_Luettersde.linkedin.com/in/luetters/

Science

Online Research Coming of age - Brownbag Presentation at Universitty of Pretoria SA