Why Pay?: Exploring How Financial Incentives are Used for Question & Answer Gary Hsieh, Robert E. Kraut, Scott E. Hudson, CHI 2010 Research Methods Data collection Mahalo Answers Rating questions in Mahalo Randomly selected 800 questions 400 were user-paid, 400 were not. Rating (1) question type (fact, opinion, advice, non-question; non-mutual exclusive), and (2) question value (sincerity, urgency, difficulty) Amazon M-Turk: 401 raters; each rated 19 questions on average Results Question asking Factual vs. is paid? Difficulty level vs. reward levels Question answering Rewards/Opinion/Sincerity vs. answer length Rewards/Opinion/Difficulty vs. answer count Opinion/Difficulty vs. answer quality Archival use Higher value; higher archival value Review Askers value longer answers, quick responses, and reliable sources Answerers learn over time (as such, receiving higher ratings, slightly earning more) Answerers tend to specialize into several topics (which may lower actual hourly wage) Financial incentives improves quantity but not quality of answers (and reputation of repliers is related to quality) Tangible (monetary) and intangible incentives have positive impact on the number of answers per answerer Question type matters (factual vs. conversational) How much a question is paid can be used as an indicator of how valuable certain Q&A exchanges are Q&A over Social Networks KSE 652 Uichin Lee Aardvark: The Anatomy of a Large- Scale Social Search Engine Damon Horowitz, Aardvark Sepandar D. Kamvar, Stanford University WWW '10 Original slides by Hailong Sun, April 13, 2010 Web Search vs. Social Q&A Google is about traditional Web search Give me keywords, I will provide contents Search for the most suitable contents While Aardvark is about social search Users can ask questions in natural language, not keywords Content is generated on-demand, tapping the huge amount of information in peoples heads (i.e., everyone knows something) The system is fueled by the goodwill of its users Problem in Aardvark How to find the user who can best answer a given question? Search Engine Aardvark Architecture Social graph indexing Users topic parsing Determine the appropriate topics for the question 23 Edit question? 1 Question? (Routing Suggestion Request) -- find a list of candidate answerers (and rank them) 3 Ask one by one 4 VarkRank: Relevance and Connectedness Topic User P(u i |t)P(u i |u j ) Used for measuring relevance score of a users question (query dependent) Used for measuring connectedness score between users (query independent) Question VarkRank: Relevance and Connectedness Given a question q, the probability that a user u i can answer it (relevance score) Score that user u i can answer a question from u j : (query dependent users query relevance score * query independent user connectedness score) Query independent user quality score = p(u i |u j ): i.e., user i delivers a satisfying answer to user j (simply measured using connectedness) 13 Relevance Scores (Expertise) How to find experts on a given topic? Expertise score: p(u i |t), and w/ Bayes law, we have: For each user, we need to profile a users interest in a given topic by using the following information sources: 3+ topics provided by a user Topics provided by friends of a user Online profiles Online unstructured data Status message updates Connectedness Scores Connection strengths between people i.e., p(u i |u j ) are computed using a weighted cosine similarity over this feature set (normalized) Utilize existing social networks Facebook, Twitter, LinkedIn Feature set measures similarities in demographics/behavior Social connection (common friends and affiliations) Demographic similarity Profile similarity (e.g., common favorite movies) Vocabulary match (e.g., IM shortcuts) Chattiness match (frequency of follow-up messages) Verbosity match (the average length of messages) Politeness match (e.g., use of Thanks!) Speed match (responsiveness to other users) Analyzing Questions: TopicMapper Question classification: Question or not? Inappropriate question? Trivial question? Location sensitive question? Map a question to a topic (weighted linear sum of the following features) Keyword matches w/ a users profile topics? Classifies the question text into a taxonomy of roughly 3000 popular question topics (using an SVM trained on an annotated corpus of several million questions) Extracting salient phrases from questions and find semantically similar user topics Ranking Algorithm Topic and connectedness matching availability Routing Engine prioritizes candidate answerers Optimize the chances that the present question will be answered Yet, preserving the available set of answerers (i.e., the quantity of answering resource in the system) as much as possible by spreading out the answering load across the user base Considering factors: currently online users (e.g., via IM presence data, iPhone usage, etc.), users daily activity history, and users response history (lowering scores of non-responsive users) Conversation Manager serially inquiring whether candidates would like to answer the present question; and iterating until an answer is provided and returned to the asker. User Interface Aardvark IMTwitter Deployment Status Aardvark is actively used Users: from 2,272 to 90,361 55.9% active users, 73.8% passive users 3,167.2 questions/day 3.1 queries/month Mobile users are particularly active Average sessions/month Comparison: Google Desktop v.s. mobile users: 3 Mobile users: 5.68 sessions/month Categories of Questions in Aardvark Questions are highly contextualized Average query length: 18.6 words 2.2~2.9 for Web search 45.3% are about context Questions often have a subjective element What are the things/crafts/toys your children have made that made them really proud of themselves? Answers 57.2% received answers in less than 10 minutes A question receives 2 answers averagely The quality of answers are good 70.4% are good; 14.1% are OK; 15.5% are bad Distribution of questions and answering timesDistribution of questions and number of answers received 21 Topic Distribution People are indexable 97.7% have 3+ topics Comparative Evaluation with Google Experimental setup Randomly select a group of questions Insert a tip do you want to help Aardvark run an experiment? Recording response time and quality of answers from Google and Aardvark Experimental results Aardvark: 71.5% answered; rating: 3.93 (=1.23) Google: 70.5% answered; rating: 3.07 (=1.46) Aardvark is more suitable for subjective questions What Do People Ask Social Networks? Meredith Ringel Morris, MSR Jaime Teevan, MSR Katrina Panovich, MIT good restaurants in atlanta Questions About Peoples Questions What questions do people ask? How are the questions phrased? What are the question types and topics? Who asks which questions and why? Which questions get answered? How is answer speed and utility perceived? What are peoples motivations for answering? What Is Known About Question Asking Collaborative search [Morris & Teevan] Searching v. asking [Evans et al.; Morris et al.] Expertise-finding [vark.com; White et al.; Bernstein et al.] Online question answering (Q&A) tools Question type [Harper et al.: conversational v. informational] Response rate [Hseih & Counts: 80%] Response time [Zhang et al.: 9 hours; Hseih & Counts: 3 hours] Motivation [Raban & Harper; Ackerman & Palen; Beenan et al.] Survey of Asking via Status Messages Survey content Used a status message to ask a question? Frequency of asking, question type, responses received Provide an example Answered a status message question? Why or why not? Provide an example 624 participants Focus on Facebook and Twitter behavior Questions About Peoples Questions What questions do people ask? How are the questions phrased? What are the question types and topics? Who asks which questions and why? Which questions get answered? How is answer speed and utility perceived? What are peoples motivations for answering? Questions: Phrasing Questions short (75 characters, 1 sentence) 18.5% of phrased as a statement I need a recommendation on a good all purpose pair of sandals. Often scoped 1 out of 5 directed to anyone Anyone know of a good Windows 6 mobile phone that wont break the bank? Network subset Hey Seattle tweeps: Feel like karaoke on the Eastside tonight? Questions: Types Type%Example Recommendation29% Building a new playlist any ideas for good running songs? Opinion22% I am wondering if I should buy the Kitchen-Aid ice cream maker? Factual17% Anyone know a way to put Excel charts into LaTeX? Rhetorical14% Why are men so stupid? Invitation9% Who wants to go to Navya Lounge this evening? Favor4% Need a babysitter in a big way tonight anyone?? Social connection3% I am hiring in my team. Do you know anyone who would be interested? Offer1% Could any of my friends use boys size 4 jeans? Questions: Topics Topic%Example Technology29%Anyone know if WOW works on Windows 7? Entertainment17%Was seeing Up in the theater worth the money? Home & Family12%So whats the going rate for the tooth fairy? Professional11% Which university is better for Masters? Cornell or Georgia Tech? Places8% Planning a trip to Whistler in the off-season. Recommendation on sites to see? Restaurants6%Hanging in Ballard tonight. Dinner recs? Current events5% What is your opinion on the recent proposition that was passed in California? Shopping5%Whats a good Mothers Day gift? Philosophy2%What would you do if you had a week to live? Missing: Health and Pornography M i s s i n g : H e a l t h a n d P o r n o g r a p h y Religion, Politics, Dating, Finance R e l i g i o n, P o l i t i c s, D a t i n g, F i n a n c e Questions: Who Asks What Type Recommendation Opinion Factual Rhetorical Invitation Favor Social connection Offer Topic Technology Entertainment Home & Family Professional Places Restaurants Current events Shopping Philosophy men women old young Twitter Facebook Questions: Motives for Asking Topic%Example Trust24.8%I trust my friends more than I trust strangers. Subjective21.5%Search engine can provide data but not an opinion. Thinks search would fail 15.2% Im pretty search engine couldnt answer a question of that nature. Audience14.9%Friends with kids, first hand real experience. Connect12.4%I wanted my friends to know I was asking the question. Speed6.6%Quick response time, no formalities. Context5.4%Friends know my tastes. Tried search5.4%I tried searching and didnt get good results. Easy5.4%Didnt want to look through multiple search results. Quality4.1%Human-vetted responses. Questions About Peoples Questions What questions do people ask? How are the questions phrased? What are the question types and topics? Who asks which questions and why? Which questions get answered? How is answer speed and utility perceived? What are peoples motivations for answering? Answers: Speed and Utility 94% of questions received an answer Answer speed A quarter in 30 minutes, almost all in a day People expected faster, but satisfied with speed Shorter questions got more useful responses Answer utility 69% of responses helpful Answers: Speed and Utility Type Recommendation Opinion Factual Rhetorical Invitation Favor Social connection Offer Topic Technology Entertainment Home & Family Professional Places Restaurants Current events Shopping Philosophy Fast Unhelpful No correlation Answers: Motives for Answering Motive%Example Altruism37.0Just trying to be helpful. Expertise31.9If Im an expert in the area. Question15.4Interest in the topic. Relationship13.7If I know and like the person. Connect13.5Keeps my network alive. Free time12.3Boredome/procrastination. Social capital10.5I will get help when I need it myself. Obligation5.4A tit-for-tat. Humor3.7Thinking I might have a witty response. Ego3.4Wish to seem knowledgeable. Motives for Not Answering - Dont know the answer - Private topic - Question impersonal M o t i v e s f o r N o t A n s w e r i n g - D o n t k n o w t h e a n s w e r - P r i v a t e t o p i c - Q u e s t i o n i m p e r s o n a l Answers About Peoples Questions The questions people ask Short, directed to anyone Subjective questions on acceptable topics Social relationships important motivators The questions that get answered Fast, helpful responses, related to length and type Answers motivated by altruism and expertise QUESTIONS? Meredith Ringel Morris Jaime Teevan Katrina Panovich M. R. Morris, J. Teevan, and K. Panovich. What Do People Ask Their Social Networks, and Why? A Survey Study of Status Message Q&A Behavior. CHI M. R. Morris, J. Teevan, and K. Panovich. A Comparison of Information Seeking Using Search Engines and Social Networks. ICWSM 2010 (to appear).