Crowdsourced Search

Victor Naroditskiy

Crowdsourced Manhunt (Before)

2

Can you find me in New York today?

Crowdsourced Manhunt (Now)

3

Tag Challenge

...and four of my friends in DC, London, Stockholm, and Bratislava.

4

Our team found 3 out of 5 and won.

5

...despite our team’s Twitter and Facebook gaining very little attention

6

Told about the challenge and our team to online

media before the competition

which led to posts on Slashdot and other forums.

7

When deciding to participate, people considered credibility of our team.

University affiliations

Winner of the Red Balloon

challenge

Mentions on CNET, ZDNET

Twitter and Facebook accounts

with updates

8

1

2

3

F

$1 for the first 2,000 referrals.

Finder receives $500.

Referrer of the finder receives

$100.

$5,000 prize distributed among participants.

9

Incentives

More details in

[Computer 2013][PLoS ONE 2013]

10

Crowdsourced Recruitingwww.thejobstack.com

11

12

DARPA Red Balloon Challenge

MIT Mechanism:½-split contract

1

2

3

F 500

62.5

250

125

recursive

13

Theoretical justification for this

mechanism.

[PLoS ONE 2012]

1

2

3

F 800

0

200

0

Invite people who may know

the answer.

finder + referrer

14

Our strategy in the Tag Challenge.

Other Ways to Motivate Recruitment

1

2

3

F 800

0

100

100Invite people who may invite many friends.

2 x finder + referrer

15

Other Ways to Motivate Recruitment

Successful paths are short (fewer than 2 referrals).

1

2

3

F 500

62.5

250

125

1

2

3

F 800

0

200

0

1

2

3

F 800

0

100

100

recursive finder + referrer 2 x finder + referrer

Use simpler non-recursive mechanisms?16

Crowdsourcing Fact-Checking

https://veri.ly/about

Is this Rome?

Crowdsourced Information Gathering

Red Balloon Challenge Tag Challenge

Successful Evidence Gathering?

Disinformation Age

Successful Fact-Checking:Crowdsourcing Critical Thinking

evidence not opinions avoid tools like “like” and upvotes for

answering questions

do not create bias by displaying popular content at the top

Pilot Experiment July 2014

25

Deployment in Nepal

Crowdsourcing Questions

Why Crowdsourcing

• People are everywhere.

• Don’t need that many people to reach the person who knows the answer:

– Tag Challenge (500 unique visitors)

– Verily Pilot (2,000 unique visitors)

crowd

sourcing

machine

learning

compute reputation

display relevant tweets/articles/images

relevant verification requests to users

submit evidence

evaluate evidence

ask friends to find the answer

Next: political fact-checking

• Real-time fact-checking at veri.ly

• Email contact@very.ly to suggest an event to fact check.

Theory: referrals and fairness

31

For every friend who joins and installs Dropbox on their computer, we'll give you

both 500 MB of bonus space.

32

1 referral is worth 1GB.

1

2

3

1/2

1/2

Recursive

+ 1/4

Invitee not compensated.

35

1

2

3

1/2

1/2

1/2

Dropbox

+ 1/2

No compensation for indirect referrals.

36

Shapley Value

37

Balanced Contributions:Your reward from my presence equals

my reward from your presence.

Efficiency:Sum of rewards = total value.

1

3

Reward: Root-dependent

38

v = 0

1

2

3

v = 2

2

3

v = 0

0

1

2

v = 1

00

0

1

2

3

1/2

1/3

1/2

Shapley

+ 1/3

+ 1/3

39

1

2

3

1/2

1/2

1/2

Symmetric

+ 1/2

+ 1/4

40

+ 1/4

Symmetric

41

Receive sd for every neighbor at distance d

Dropbox: s1 = 1/2, s2 = 0, s3 = 0 ...

Satisfies balanced contributions!

Symmetric vs Geometric

43

Setting sd = (1/2)d makes Symmetric similar to Geometric

Compensation for descendants is the same under Geometric and Symmetric, but Symmetric also

compensates for ancestors

Geometric mechanisms do not satisfy balanced contribution

1

3

Reward: Root-independent

44

1

2

3

v = sum of rewards

2

3

s1

1

2

s1s2

s2

Symmetric vs Geometric

Recursive Shapley

No balanced contributions.

Balanced Contributions:

Root-dependent

45

1

2

3

1/2

1/3

1/2 + 1/3

+ 1/3 1/2

1/2

1/2 + 1/2

+ 1/4

+ 1/4

Symmetric

Balanced Contributions

1/2

0

1/2

+ 1/4

Recursive Shapley

46

1

2

3

833

333

833

600

600

800

Symmetric

1200

0

800

Normalized to Distribute 2GB

Trees

47

Referral structure:Under all mechanisms, an ancestor is compensated for at

least as many nodes as his descendent.

1

2 3

4 5 6 7

8 9 10

Thank you

48

49

Extra Slides

Crowdsourcing Marketing

• Word-of-Mouth referrals

Crowdsourcing Marketing

• Appealing to organisations with low marketing budgets– Charities

– Crowdfunding

– Volunteer Crowdsourcing

• Can we incentivisereferrals here?– Direct monetary incentives not appropriate

– Gamification?

Field Study

• Cancer Marathon

Field Study

Field Study

Adding Incentives

Treatments

• Control

– No additional incentive

• Low

– 1 additional point

• High

– 3 additional points

• Recursive

– 1 for direct, ½ for referral of referral, etc.

Treatments

• Control

– No additional incentive

• Low

– 1 additional point

• High

– 3 additional points

• Recursive

– 1 for direct, ½ for referral of referral, etc.

Treatments

• Control

– No additional incentive

• Low

– 1 additional point

• High

– 3 additional points

• Recursive

– 1 for direct, ½ for referral of referral, etc.

+1 +1

Treatments

• Control

– No additional incentive

• Low

– 1 additional point

• High

– 3 additional points

• Recursive

– 1 for direct, ½ for referral of referral, etc.

+3 +3

Treatments

• Control

– No additional incentive

• Low

– 1 additional point

• High

– 3 additional points

• Recursive

– 1 for direct, ½ for referral of referral, etc.

+1 +1

+0.5 +0.5

+0.25

Questions

• Do points make any difference in generating referrals?

• Does the quantity of points make a difference?

• Is the recursive mechanism effective?

Evaluation

• 17,686 students emailed

• 1,042 unique users visited website

• 412 signed up

• 65 took sharing action

• 46 generated at least one visit (456 visits)

• 19 generated at least one sign-up (37 sign-ups)

Visit Sign-upSharing Action

Referred Visit

Referred Sign-up

Evaluation

Differences in Referred VisitsU

sers

Refe

rrin

g 1

+

Vis

its

Differences in Referred Sign-upsU

sers

Refe

rrin

g 1

+ S

ign-

Ups

Channels Used

Referred Visits: Referred Sign-ups:

Results Summary

• Referred Visits:

– High > Control, High > Low

• Referred Sign-ups:

– High > Control, High > Recursive

Conclusions

• Referral incentives make a difference.

• Magnitude of incentives is important.

• Enables novel fundraising drives:

– Release donation gradually to increase impact.

Results

Extensive Margin Results