47
Precision Performances. Uncompromising Results.

Precision Performances. Uncompromising Results

Embed Size (px)

Citation preview

Page 1: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Page 2: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

B2B Internet Research:the Key to SuccessRoland Klassen

[email protected]

Tim Sinke

[email protected]

Page 3: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Who are we andwhat’s our bias?

Page 4: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Acrobat Research• CATI and Web – 3 offices – Toronto, Sudbury,

Canso, NS

• Web since 1997

• Majority of Business is CATI

• Background is B2B, traditional focus in IT

• Approach has been to attempt to take an impartial view

Questions to Ask

Page 5: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Questions to Ask

Page 6: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Population to be Sampled• Identifiable?

• Online?

• Reachable on phone?

• Over-surveyed?

• Already exists in a panel?

Questions to Ask

Page 7: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Type of Info to be Collected?• Complexity?

• Length?

• Stimuli needed?

• Mixed-mode?

Questions to Ask

Page 8: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Geographical coverage?

Questions to Ask

Page 9: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Does it need to be Projectable?• Limitation based on company

size/industry?

• Does it have to be matched to D&B SICs/NAICs/etc?

Questions to Ask

Page 10: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Other Questions• Response Rate

• Time in field

• Incentives

• Budget Expectations

– Faster and cheaper, right?• Maybe

Questions to Ask

Page 11: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Other Questions...

• CAN the survey be self-administered?

• SHOULD the survey be self-administered?

Questions to Ask

Page 12: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Sample Sources

Page 13: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Internal Lists

• Customers

• Employees

• Prospects

Sample Sources

Page 14: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Drop-in/Intercepts• Web-site specific

• For general web sample – dropping – click-through rates are dropping off to nothing

Sample Sources

Page 15: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Online Panels• Main general panels in NA are SSI,

Greenfield, e-Rewards, Harris Interactive

– Get specifics on their panel build and profile processes

• Specialty panels

Sample Sources

Page 16: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

A word on major online panels

• Educate yourself

• Review your screening criteria

• Are the numbers available?

• Geographies

Sample Sources

Page 17: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Who are you talking to?• Went to www.ask.com – typed “online

market research pros and cons” got these…

Sample Sources

Page 18: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Sample Sources

Page 19: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Sample Sources

Page 20: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

A Quick Review of one Player • Used SSI as an example

– Not meant to support or criticize

• Following are excerpts from their website:

Sample Sources

Page 21: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

What it says:• SurveySpot panelists are recruited from many sources. These

include banner ads on the Web and other Internet methods, and by RDD telephone. All SurveySpot members are recruited exclusively by permission-based techniques. SSI does not use unsolicited e-mail (or spam) in building the panel.

• SurveySpot panel demographics are not based on predictive techniques. They are created from self-reported, respondent-specific information. Researchers are assured that they are reaching the exact target market they seek meaning more cost-effective online research [EMPHASIS ADDED]

Sample Sources

Page 22: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Industry Select Offered:• Industry

– Agriculture SIC 01-09– Mining SIC 10-14– Contracting SIC 15-17– Manufacturing SIC 20-39– Utilities SIC 40-49– Wholesale SIC 50-51

Sample Sources

– Retail SIC 52-59– Financial SIC 60-67– Services SIC 70-79– Professionals SIC 80-89– Government SIC 91-97

Page 23: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

What are the issues with this?• Respondents don’t know their SIC Code

• How are SIC Codes mapped and assigned? Primary/secondary? To what level of detail?

• Does it matter to you?

• Do they match back to D&B if you’re extrapolating to the industry?

Sample Sources

Page 24: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Recruit-to-Web

Page 25: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Background• What is R2W?

– CATI recruit – web completion– User ID/Password protected– Appropriate follow ups – based on

response/completion stage

• Why use R2W?

– Can’t reach everyone through a panel– You want to be sure of the respondent

Recruit-to-Web

Page 26: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Examples• Most senior C-Level IT person in Global 500

corporations – US and Europe

• Office equipment DMs mailed physical samples to take a follow-up survey

• Telecom decision makers recruited in cross-section of businesses in specific geographies

• Results – in Survey Stats segment

Recruit-to-Web

Page 27: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

General Process• (CATI) Recruit

– Custom email – immediate or batch– Follow up – 48-72 hours– Follow up – 1 week– Optional after that

Recruit-to-Web

Page 28: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Applications• Can be effective; however

• Use to

– Support definite online requirement – can’t be done on the phone

– Refusal conversion and/or multi-modal survey• If “standard” tool – could become more expensive

than CATI

• More rigorous in defining and controlling the respondent

Recruit-to-Web

Page 29: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Survey Stats

Page 30: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Customer/Internal Lists• Response rates – 5 to 75%

• Lower end tends to be lower than low end of CATI CSAT

• What is less intrusive – email (suspected SPAM) or CATI (suspected telemarketing)?

– Subject line/intros/”warm-ups” are key

Survey Stats

Page 31: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Web Panel• Cost on CPI – response rate “isn’t your

problem” – however, feasibility is

• Be careful on over-selling

Survey Stats

Page 32: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Factors Influencing Response Rates

• Survey Length

• Respondent Level

• Incentive

• Reminders

• Company Size

• External stimuli/downloads required – obstacles

• Geography

Survey Stats

Page 33: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Examples of Survey Response• Historical

– Used to target/achieve 50-60% conversion on R2W studies

– Declining – now aggressive target is 35-50%

Survey Stats

Page 34: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

C-Level in Global 500 Corporations

• R2W – High Incidence; most senior contact; 40 minutes, $300 US incentive. Conversion – US – 42.8%; Euro 16.9%

Survey Stats

367

15791 14

125

2459

9

325

0 100 200 300 400 500 600

US

UK

IT

GE

FR

Page 35: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

International Office Equipment Study

• R2W – High Incidence; 30 minutes, incentive $50 US; 40 Euros (had to scale up to 100 Euros). Conversion – US – 58.2%; Euro 20.9%

Survey Stats

569 327

402 91

244 65

258

33

0 200 400 600 800 1000

US

UK

GE

FR

Page 36: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Production Environment Study

• Web Panel - Europe

Survey Stats

Page 37: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Telecom DM (internet)• R2W – Small bus Telecom DMs; 51 minutes $35

incentive. Conversion 35.5%

Survey Stats

685

243

0 100 200 300 400 500 600 700

Recruits

CPs

Page 38: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

A Different View• From e-Rewards.

• Demographic Profile Variable analysis of response rates, based on:

– Tot emails sent: 5.7 m– Tot surv resp: 1.3 m

(NOTE: Not sent on B2B studies specifically)

Survey StatsOccupation Resp Rate

Exec/Upper Mgt 17.9%

Sales 18.8%

Military 20.2%

Bus Owner/Self Empl 21.6%

Crafts/Constr 21.8%

Farming/Primary 23.0%

MEAN 23.0%

Mid Mgt 23.2%

Prof/Tech 24.3%

Retired 24.5%

Student 24.8%

Other 25.0%

Artistic/Literary 25.7%

Clerical 26.4%

Unemployed 27.0%

Doctor 27.9%

Teacher 30.2%

Homemaker 30.3%

Page 39: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Quality of Data• Lower response rates – higher bias potential?

• Who are the non-responders?

• Equally important – who are the responders?

– Who’s answering – quality/validation checks?

• Web response rates declining, but not yet worse than telephone or mail

Survey Stats

Page 40: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Incentives

Page 41: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

What’s required – how much and how?

• Panels – costs included in CPI – may change for B2B

• Pressure is increasing

Incentives

Page 42: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Amounts to Pay• Low-level - $1 per

minute

• Mid-level - $1.50-$2.00 per minute

• High-level - $35+ (short); $40+ (mid); $50+ (long) – add $2.00 per minute above 20 minutes in length

Incentives

0

5

10

15

20

25

30

35

40

45

50

Low Level Mid Level High Level

Short (<10 min) Med (15 min) Long (20+min)

Page 43: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

How to Pay• Cash (cheque)

• E-Certificate (simpler, but expectation of instant delivery)

• Close survey and review data prior to issuing incentives

• Notify respondents of timeframes

• Create a centralized database and 800 number/central email for queries

Incentives

Page 44: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

KSF – Summary Checklist

Page 45: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Check points for Success (1)• Question research objectives

• Multi-modal?

• Pick sample partner carefully

• Put sample partner “through the wringer”

• Analyze data for comparison and validation

KSF – Summary Checklist

Page 46: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Check points for Success (2)

• Are respondents engaged? Devise tests

– Look for repetition– Average length– Volume of DK/RF

• Work in partnership with data collection partner

• Be ready to adjust on the fly

• Always, always, always pilot qualitatively

• Remember, you’re only as good as your sample

KSF – Summary Checklist

Page 47: Precision Performances. Uncompromising Results

Precision Performances. Uncompromising Results.

Thank you. Questions?