Differential Sampling Based on

Historical Individual-Level Data in

Online Panels

By Richard Kelly

Overview

The growth in the market has been rapid, and while methodological improvements have come fairly quickly, online survey research is not yet mature.

Online sampling can be improved by supplanting traditional marginal population quotas for a balanced population entering the survey by employing differential sampling

Online Research

Disadvantages

Advantages

Growth in online survey research is not

likely to ebb anytime soon

Still, there remains room for improvement

The survey “research industry is wedded

to quota controls, sometimes to the point

of obsession. It places them on Age and

Gender without a second thought as to

why or indeed whether they are doing any

good at all.” (Cape, 2011)

Researchers who do not know the

demographic composition of the

population they are attempting to study

should not enforce general population

quotas in an attempt to yield

“representative” data

However, we should not abandon controls

on demographics

Population quotas

force something

artificial onto a

sample group.

Differential Sampling

The sample entering the survey mimics the

population under study…each respondent has

an equal opportunity to enter the research effort

and is able to screen out in their natural

incidence.

optimizes sampling efficiency

Advantages of online access panels

Individual historical propensity scores

Determining invitations by demographic group

Test

Ran two surveys identical in content

Survey A = population quotas

Survey B = differentially sampled

Respondents screened on 12 month

purchase intent for auto insurance

Expectations

Demographics in final data would be

substantially different from Survey A to

Survey B

Survey health (incidence) of Survey A

would decline relative to Survey B

Significant difference in data between

Survey A and Survey B

ResultsSurvey A Survey B

Final Quota Target Final Actual Return Return Target

Male 49 49 62 50% 49%

Female 51 51 38 50% 51%

18-24 13 13 12 15% 13%

25-34 18 18 17 17% 18%

35-49 28 28 25 28% 28%

50+ 41 41 46 40% 41%

Northeast 18 18 11 17% 18%

Midwest 22 22 20 20% 22%

South 37 37 44 39% 37%

West 23 23 25 25% 23%

$29,999 or less 31 31 20 31% 31%

$30,000-$49,999 19 19 22 19% 19%

$50,000-$74,999 18 18 24 20% 18%

$75,000 or more 32 32 34 30% 32%

Results (con’t)

Survey Health

Survey A final incidence = 21.8%

Survey B final incidence = 28%

Results (con’t)

Satisfaction with current auto insurance provider

Survey A mean score: 4.06

Survey B mean score: 3.99(not statistically significant)

Average annual spend (ordinal scale)

Survey A mean score: 2.5

Survey B mean score: 2.84p<.05

Conclusions

Marginal population quotas in online

research are a persistent holdover from

other sampling frames that don’t

necessarily apply to a different frame.

Issues of representivity can be mitigated

by using differential sampling instead of

population quotas.

Much more future research is needed