View
23
Download
3
Category
Tags:
Preview:
DESCRIPTION
Crowd Crawling: Towards Collaborative Data Collection for Large-scale Online Social Networks. Cong Ding, Yang Chen*, and Xiaoming Fu University of Göttingen *Duke University. Significance of social network data crawling. Understanding user behaviors Improving SNS architectures - PowerPoint PPT Presentation
Citation preview
Crowd Crawling: Towards Collaborative Data Collection for Large-scale Online Social Networks
Cong Ding, Yang Chen*, and Xiaoming Fu
University of Göttingen*Duke University
Significance of social network data crawling
•Understanding user behaviors
•Improving SNS architectures
•Handling privacy/security issues
•and so on...
Current data collection methods (1)
•ISP-based measurement [Schneider IMC’09]
Only ISP companiescan do that
Current data collection methods (2)
•Cooperate with SNS companies [Yang IMC’11]
Most research groupsdo not have chance
Current data collection methods (3)
•Crawl data by a single group (and share them to others)
[Gjoka INFOCOM’10]
Suffering requestrate limiting
Shortages of crawling by a single group
•Waste computing andnetwork resources
•Introduce overhead toservice providers (andmay lead stricter rate limiting)
•Lack of ground truth forthe research community
A new thought
Why not collect data collaboratively?
System overview
Coordinator
Crawlers
System design
•Fetching UIDs (BFS, etc.)
•Handling crawling failure (timeout)
•Bypassing request rate limiting (massive IP addresses)
•Data fidelity (redundant crawling)
Implementation
•A proof-of-concept prototype (without the data fidelity part)to crawl in Weibo
•472 PlanetLab servers as crawlers
Evaluation
•In 24 hours, we have crawled 2.22M users’ data from Weibo,including user profiles, all the posts, all the social connections
•Comparison:
•Fu et al. (PLOS ONE 2013) get 30K user’s data in 6 days•Guo et al. (PAM 2013) get 1M user’s data in 1 monthCrowd
CrawlingFu et al. Guo et al.
#UIDs/day 2.22M 5K 33K
Evaluation
Evaluation
Conclusion and Discussion
•Data sharing may violate some providers’ terms of servicesoTwitter does not allow to share data (even for
research)oWeibo allows to share data among researchers
•Unlimited data sharing might cause ethical issuesoThe data should be anonymized
•We will publish the data crawled in the evaluation
Recommended