View
2
Download
0
Category
Preview:
Citation preview
Crowdsourced Search
Victor Naroditskiy
Crowdsourced Manhunt (Before)
2
Can you find me in New York today?
Crowdsourced Manhunt (Now)
3
Tag Challenge
...and four of my friends in DC, London, Stockholm, and Bratislava.
4
Our team found 3 out of 5 and won.
5
...despite our team’s Twitter and Facebook gaining very little attention
6
Told about the challenge and our team to online
media before the competition
which led to posts on Slashdot and other forums.
7
When deciding to participate, people considered credibility of our team.
University affiliations
Winner of the Red Balloon
challenge
Mentions on CNET, ZDNET
Twitter and Facebook accounts
with updates
8
1
2
3
F
$1 for the first 2,000 referrals.
Finder receives $500.
Referrer of the finder receives
$100.
$5,000 prize distributed among participants.
9
Incentives
More details in
[Computer 2013][PLoS ONE 2013]
10
Crowdsourced Recruitingwww.thejobstack.com
11
12
DARPA Red Balloon Challenge
MIT Mechanism:½-split contract
1
2
3
F 500
62.5
250
125
recursive
13
Theoretical justification for this
mechanism.
[PLoS ONE 2012]
1
2
3
F 800
0
200
0
Invite people who may know
the answer.
finder + referrer
14
Our strategy in the Tag Challenge.
Other Ways to Motivate Recruitment
1
2
3
F 800
0
100
100Invite people who may invite many friends.
2 x finder + referrer
15
Other Ways to Motivate Recruitment
Successful paths are short (fewer than 2 referrals).
1
2
3
F 500
62.5
250
125
1
2
3
F 800
0
200
0
1
2
3
F 800
0
100
100
recursive finder + referrer 2 x finder + referrer
Use simpler non-recursive mechanisms?16
Crowdsourcing Fact-Checking
https://veri.ly/about
Is this Rome?
Crowdsourced Information Gathering
Red Balloon Challenge Tag Challenge
Successful Evidence Gathering?
Disinformation Age
Successful Fact-Checking:Crowdsourcing Critical Thinking
evidence not opinions avoid tools like “like” and upvotes for
answering questions
do not create bias by displaying popular content at the top
Why Crowdsourcing
• People are everywhere.
• Don’t need that many people to reach the person who knows the answer:
– Tag Challenge (500 unique visitors)
– Verily Pilot (2,000 unique visitors)
crowd
sourcing
machine
learning
compute reputation
display relevant tweets/articles/images
relevant verification requests to users
submit evidence
evaluate evidence
ask friends to find the answer
Next: political fact-checking
• Real-time fact-checking at veri.ly
• Email contact@very.ly to suggest an event to fact check.
Theory: referrals and fairness
31
For every friend who joins and installs Dropbox on their computer, we'll give you
both 500 MB of bonus space.
32
1 referral is worth 1GB.
1
2
3
1/2
1/2
Recursive
+ 1/4
Invitee not compensated.
35
1
2
3
1/2
1/2
1/2
Dropbox
+ 1/2
No compensation for indirect referrals.
36
Shapley Value
37
Balanced Contributions:Your reward from my presence equals
my reward from your presence.
Efficiency:Sum of rewards = total value.
1
3
Reward: Root-dependent
38
v = 0
1
2
3
v = 2
2
3
v = 0
0
1
2
v = 1
00
0
1
2
3
1/2
1/3
1/2
Shapley
+ 1/3
+ 1/3
39
1
2
3
1/2
1/2
1/2
Symmetric
+ 1/2
+ 1/4
40
+ 1/4
Symmetric
41
Receive sd for every neighbor at distance d
Dropbox: s1 = 1/2, s2 = 0, s3 = 0 ...
Satisfies balanced contributions!
Symmetric vs Geometric
43
Setting sd = (1/2)d makes Symmetric similar to Geometric
Compensation for descendants is the same under Geometric and Symmetric, but Symmetric also
compensates for ancestors
Geometric mechanisms do not satisfy balanced contribution
1
3
Reward: Root-independent
44
1
2
3
v = sum of rewards
2
3
s1
1
2
s1s2
s2
Symmetric vs Geometric
Recursive Shapley
No balanced contributions.
Balanced Contributions:
Root-dependent
45
1
2
3
1/2
1/3
1/2 + 1/3
+ 1/3 1/2
1/2
1/2 + 1/2
+ 1/4
+ 1/4
Symmetric
Balanced Contributions
1/2
0
1/2
+ 1/4
Recursive Shapley
46
1
2
3
833
333
833
600
600
800
Symmetric
1200
0
800
Normalized to Distribute 2GB
Trees
47
Referral structure:Under all mechanisms, an ancestor is compensated for at
least as many nodes as his descendent.
1
2 3
4 5 6 7
8 9 10
Thank you
48
49
Extra Slides
Crowdsourcing Marketing
• Word-of-Mouth referrals
Crowdsourcing Marketing
• Appealing to organisations with low marketing budgets– Charities
– Crowdfunding
– Volunteer Crowdsourcing
• Can we incentivisereferrals here?– Direct monetary incentives not appropriate
– Gamification?
Field Study
• Cancer Marathon
Field Study
Field Study
Adding Incentives
Treatments
• Control
– No additional incentive
• Low
– 1 additional point
• High
– 3 additional points
• Recursive
– 1 for direct, ½ for referral of referral, etc.
Treatments
• Control
– No additional incentive
• Low
– 1 additional point
• High
– 3 additional points
• Recursive
– 1 for direct, ½ for referral of referral, etc.
Treatments
• Control
– No additional incentive
• Low
– 1 additional point
• High
– 3 additional points
• Recursive
– 1 for direct, ½ for referral of referral, etc.
+1 +1
Treatments
• Control
– No additional incentive
• Low
– 1 additional point
• High
– 3 additional points
• Recursive
– 1 for direct, ½ for referral of referral, etc.
+3 +3
Treatments
• Control
– No additional incentive
• Low
– 1 additional point
• High
– 3 additional points
• Recursive
– 1 for direct, ½ for referral of referral, etc.
+1 +1
+0.5 +0.5
+0.25
Questions
• Do points make any difference in generating referrals?
• Does the quantity of points make a difference?
• Is the recursive mechanism effective?
Evaluation
• 17,686 students emailed
• 1,042 unique users visited website
• 412 signed up
• 65 took sharing action
• 46 generated at least one visit (456 visits)
• 19 generated at least one sign-up (37 sign-ups)
Visit Sign-upSharing Action
Referred Visit
Referred Sign-up
Evaluation
Differences in Referred VisitsU
sers
Refe
rrin
g 1
+
Vis
its
Differences in Referred Sign-upsU
sers
Refe
rrin
g 1
+ S
ign-
Ups
Channels Used
Referred Visits: Referred Sign-ups:
Results Summary
• Referred Visits:
– High > Control, High > Low
• Referred Sign-ups:
– High > Control, High > Recursive
Conclusions
• Referral incentives make a difference.
• Magnitude of incentives is important.
• Enables novel fundraising drives:
– Release donation gradually to increase impact.
Results
Extensive Margin Results
Recommended