Can We Effectively Sample from Social Media Sites and ... · Facebook, Google Adwords, and Google...

Results from Two Sampling Experiments

Michael J. Stern, PhD, Kirk Wolter, PhD,and Ipek Bilgen, PhD.

Can We Effectively Sample from SocialMedia Sites and Search Engines?

…"interactive platforms via which individuals ….. create and shareuser-generated content” (see Kietzmann, Hermkens, McCarthy,and Silvestre, 2011).

…use of search engines and ads.

• Why has there been a call for exploring this research?– Increasing difficulty reaching respondents; Expense of pre-testing

instruments; Interest and recent, tepid acceptance of non-probabilitymethods; Client expectations; Mobile devices lend themselves; andPanel construction.

What is Social Media?

Active versus Passive Social Media and Search Engine DataCollection

• Active (we’ll be focusing here today)• Using social media sites and search engines to collect sample,

collect survey responses, or both.

• Passive• Using social media sites and search engines to collect data

through scraping or other systematic techniques that allow foranalyzing aggregate user-generated data or paradata.

Types of Social Media and Search Engine Surveying

Selected Examples

• Active• In Bhutta’s (2012) study of baptized Catholics, she created a

Facebook group named “Please Help Me Find BaptizedCatholics!” and then contacted the administrators from otherCatholic-centered Facebook groups to recruit members for herstudy.

• Ramo and Prochaska (2012) used Facebook to recruitparticipants for a study of cigarette users between the ages of 18-25 years old. Within three months, they had obtained a sample of3,093 individuals who were eligible for the study with 1,548completing the survey at a cost of $4.28 per complete.

Examples of previous work

Example

• Passive• One example is the Google

Flu Trend data, whichoverestimated prevalencethis year.

Examples of previous work

Used By Permission--Butler, 2013Sources: Google Flu Trends (www.google.org/flutrends); CDC; FluNear You

Example

• Passive• Kosinski et al (2013)

analyzed “likes” onFacebook for 58Kvolunteers. Using thisdata they accuratelypredicted, to name afew:

Single/In Rel.: 67%Sex: 93%Race: 95%Political Affil.: 85%Sexual Orien.: 88%

Examples of previous work

Kosinski, Stillwell, and Graepel. (2013) “Private traits and attributesare predictable from digital records of human behavior.”www.pnas.org/cgi/doi/10.1073/pnas.1218772110

1. What is this data good for? Augment Supplement Primary All of the above

The question rests in how representative it is.

Questions that we must address

• To address this question, we must examine how it works

• Bids (suggested/who sees the ad/targeting)

• Measuring Response Rates

• Click Through Rates—The number of unique clicks you receiveddivided by the number of times your ad was shown.

• Clicks versus Visits versus Responses—examples fromFacebook, Google Adwords, and Google Analytics

How does active social media surveying work?

• Experiments• Design Facebook Ads (today n=134)• Google Ad (today n=384)• Both benchmarked by 2010 General Social Survey (2,041)

• Other• ABS and Email Blast (see Bilgen, Wolter, and Stern, 2013)• Purpose• Representativeness/weighting/effective recruitment/cost per case

• Incentives• $10• $5• $2

• Programming challenges• PINs and No PINs

• Three prong system using cookies, IP addresses, and email registration

Our Work in Active Data Collection

What’s a response rate based on?

Click throughby Reach

Is this theresponse rate?

Finding and recruiting respondents

We used avariety of adsas a way torecruitrespondents toour survey.Each adperforms at asomewhatdifferent leveland real timeanalytics foreach.

Finding and recruiting respondents

We included theincentive value($10 or $5depending ontreatment)

And, ouraffiliation.

Clicking on thead takesrespondents toour landing page

Finding and recruiting respondents through Google

We included the incentive value ($10or $5 depending on treatment), ouraffiliation. Clicking on the ad takesrespondents to our landing page

Google Adwords requires keywords

Differences by approach? Age

51.7

11.8

0.1 0.1 0.1

18-24 years old 25-34 years old 35-49 years old 50-64 years old 65 or older

Age: Facebook Analytics-who clicked the ad?

Facebook Analytics (%)

Just over a third of cases in analytics were missing ageor sex

Differences by approach? Age

25.4

7.6

16.9

33.9

13.6

51.7

11.8

0.1 0.1 0.1

18-24 years old 25-34 years old 35-49 years old 50-64 years old 65 or older

Age: Facebook Analytics and Survey Results

Facebook Survey Data (%) Facebook Analytics (%)

Obviously, the 0.1 folks don’t include their age, sex or both…andrespond at higher rates

Differences by approach? Age

8.9

18.5

26.9 26.6

19.3

25.4

7.6

16.9

33.9

13.6

10.4

20.8

39.0

21.7

8.2

18-24 years old 25-34 years old 35-49 years old 50-64 years old 65 or older

Variation in Age Distribution: GSS, Facebook, and GoogleGSS Facebook Google FB vs. Google

Chi-Square= 46.52; p <0.001

Differences by approach? Sex

54.9

45.1

51.246.1

Male Female

Age: Facebook Analytics and Survey ResultsFacebook Survey Data (%) Facebook Analytics (%)

Differences by approach? Sex

43.6

56.454.9

45.142.9

57.1

Male Female

Variation in Sex Distribution: GSS, Facebook, and GoogleGSS Facebook GoogleFB vs. Google

Chi-Square= 5.02; p =0.03

Differences by approach? Education

63.9

25.5

10.7

47.2

39.2

13.6

62.2

25.9

11.5

Less than a College Degree College Degree Advanced Degree

Variation in Educational Attainment Distribution: GSS,Facebook, and GoogleGSS Facebook Google

FB vs. GoogleChi-Square= 49.23; p =0.01

Differences by approach? Work Status

44.9

11.5

35.8

4.6

32.5

15.4

48.0

4.1

37.6

16.9

31.4

14.1

Full Time Part Time Not employed In School

Variation in Work Status: GSS, Facebook, and GoogleGSS Facebook Google FB vs. Google

Chi-Square= 15.14; p =0.002

• Facebook and Google• Generalizability? Questionable but Google outperformed

Facebook.

• Appropriateness may hinge on sample requirements

Our Work in Active Data Collection

• Cost Per Case and Spending?

Our Work in Active Data Collection

FACEBOOK SPENT(DATA COLLECTION—does not include incentive )

4,215 total clicks

123 completes

34.27 clicks per complete

$0.87 spent per click (average)

$29.98 spent per complete

$3,687.04 TOTAL Spending

GOOGLE SPENT(DATA COLLECTION—does not include incentive )

5,127 total clicks

318 completes

16.12 clicks per complete

$0.79 spent per click (average)

$12.09 spent per complete

$3,846.00 TOTAL Spending

• Google looks more like GSS than Facebook.

• Google was faster and cheaper.

• Had one security blip with Google, though, withslickdeals.net

Ad Campaigns

• What have clients asked for

• Other uses

• What we’re doing

• IRB

Thinking Ahead

Thank You!

Please contact me with any questions or comments:stern-michael@norc.org