Understanding Google Answers

KSE 801Uichin Lee

Outline

• Earnings and Ratings at Google Answers (Edelman 2004)– General statistics, what do askers value, experience and

learning on the jobs, hourly wages, specialization• Knowledge Market Design: A Field Experiment at

Google Answers, (Chen 2008) – Effects of prices, tips, and reputation on answer quality and

answerer’s effort• The Incentive Structure in an Online Information

Market (Raban 2008)– Effects of monetary/non-monetary incentives

• Online knowledge market offered by Google that allowed users to post bounties for well researched answers to their queries.

• Launched in April 2002 and came out of beta in May 2003. • Asker-accepted answers cost $2 to $200. • Google retained 25% of the researcher (=answerer)'s reward and a 50

cent fee per question. • In addition to the researcher's fees, a client who was satisfied with the

answer could also leave a tip of up to $100. • In late November 2006, Google reported that it planned to

permanently shut down the service, and it was fully closed to new activity by late December 2006, although its archives remain available

• GA received more than 100 question postings per day when the service ended in December 2006.

From Wikipedia: http://en.wikipedia.org/wiki/Google_Answers

• Researchers (called Google Answers Researchers or GARs) – Not Google employees, but contractors that were required to complete an

application process to be approved to answer for the site.– GARs were limited in number (~500). – Application process tested their research and communication abilities.– Researchers with low ratings could be fired, a policy which encouraged

eloquence and accuracy. • For a Researcher, a question was answered by logging into a special

researchers page and then "locking" a question they wanted to answer. • This act of "locking" claimed the question for that researcher.

– Questions worth less than $100 could be locked for up to four hours, – Questions worth more than $100 could be locked up to eight hours at a time in

order to be properly answered. – A Researcher could only lock one question at a time (or two)

From Wikipedia: http://en.wikipedia.org/wiki/Google_Answers

Earnings and Ratings at Goggle Answers

Benjamin EdelmanHarvard Business School Economic Inquiry, 2004

Descriptive Stats of GA

43% answered63% rated answers21% tipped answers

The Incentive Structure in an Online Information Market (Raban 2008)

(Edelman 2004)April 2002- Nov. 2003

Number of Answers per GAR(Google Answer Researcher)

• Approximately follows a power-law distribution

Frequent/Active GARs

Occasional/Less Active GARs

The Incentive Structure in an Online Information Market (Raban 2008)

Time Difference: Posting and Answer

(mins) (mins)

10000 minutes equals to 6.9 days (expires after a week)

1440 minutes equals one day480 minutes equals to 8 hours

What Do Askers Value?

• Rate, tips vs. answer length, # URLs, time-spent (for answers)

• Longer answers elicit better rating/tips (slight improvement though)

• # URLs and answer length highly correlated• Asker who waited longer for answer is less

likely to assign a top rating

Experience and Learning on the Job

• Labor market literature: on-the-job learning plays a significant role in developing worker skills and facilitating worker productivity (Jovanovic and Nyarko 1996)

• Answer with 10 or more questions is 0.28% more likely to be rated a 5 (and w/ more 2.5 cents more tips)

• Selection effect: higher quality answerers enjoy higher ratings and elect to participate more or longer

• Experience users paid by $7.63 per hour (but only showing $0.02 difference based on experience)

Specialization

• Specialization results in less payment per hour • Answerers may forgo the opportunities in

other fields (as there aren’t many questions available to the answerers)

• Askers feel better when they get answers from “experts” (giving favorable ratings)

Day of Week, Hour of Day

• Relative lack of questions over the weekend • Mondays have the highest pay per minute

(relative lack of answerers on Mondays)• Questions posted from 8-10PM get fastest

responses• Answers made during business hours receive

better ratings/tips (for tips, it’s 15% vs. 8%)

Knowledge Market Design: A Field Experiment at Google Answers

Yan Chen, Teck-Hua Ho, Yong-Mi Kim School of Information

University of MichiganJournal of Public Economics Theory, 2010

Understanding Answer Quality• Hypothesis 1 (Reciprocity: effort)

– A question with a higher price generates an answer involving more effort.• Hypothesis 2 (Reciprocity: quality)

– A question with a higher price generates an answer with better quality.• Hypothesis 3 (Tip: effort)

– A conditional (or unconditional) tip generates an answer involving more effort compared to a fixed price $20 question.

• Hypothesis 4 (Tip: quality)– A conditional (or unconditional) tip produces a higher quality answer than a

fixed price $20 question.• Hypothesis 5 (Unconditional vs. Conditional Tips)

– An unconditional tip produces a better answer than a conditional tip.• Hypothesis 6 (Reputation)

– Answerers with higher reputation will provide better answers.

Methods

• Treatment settings:– $20 fixed price– $30 fixed price– $20 plus an unconditional $10 tip– $20 plus a conditional $10 tip

• 100 question-answer pairs from IPL (Internet Public Library), a non-profit organization that provides a question-answering reference service– At least taking 10-30 minutes of work by knowledge

workers

Methods

• For each question-answer pair, human raters provide ratings using the following metrics: – Difficulty (1: very easy, 5: very difficult)– Factors(1: strongly disagree, 5: strongly agree, N/A):

• The question that was asked is answered.• The answer is thorough, addressing all question parts.• The sources cited are credible and authoritative.• The links provided are to relevant web sites or pages.• Information in the cited sources is summarized.• Only information pertinent to the question is presented.• The answer is well-organized and written clearly, avoiding jargon and/or

inappropriate language

– Quality of the answer (1: very low, 5: very high)

Methods

• Information science students rated the QA pairs– 100 IPL questions used for GA experiments– 25/100 GA question-answer pairs– When compared ratings with asker’s ratings from Google

Answers, they tend to give lower ratings but the correlation is very high (0.847)

• Sent 100 IPL questions to Google Answers in July, Oct., Nov. 2005– Each day 4 questions were sent to GA (one from each

treatment)• Results: total 76 answered out of 100 questions

Results

# questions answered

Higher price leads to longer answers

Answerer’s w/ higher reputation provides longer answers

Results

Answerer’s w/ higher reputation provides better answer quality

Tip does not make any significant influence

Higher prices does not make any significant influence (very different from Harper et al. 2008)

The Incentive Structure in an Online Information Market

Daphne Ruth RabanJournal of the American Society for Information

Science and Technology Volume 59, Issue 14, pages 2284–2295, December 2008

Incentives• Intangible incentives:

– Intrinsic motivation; e.g., self-efficacy (just activity itself), altruism (enjoyment of helping others), warm glow (donation)

– Ratings, conversation (comments), expansion of social capital• Tangible incentives:

– Monetary or economic incentives• Tale of two markets (Heyman and Ariely 2004):

– Monetary markets: output proportional to monetary incentives– Social markets (altruism): output is insensitive to incentives (e.g., candy)– Mixed markets tend to follow “monetary markets” (e.g., once candy’s

price is revealed, it acts like monetary markets)• Crowding-out effect (Frey & Goette, 1999): monetary incentive reduced the

motivation to do volunteer work

Incentives• Linkage between motivations/incentives and willingness to share

knowledge – Intrinsic motivation for knowledge sharing within organization (Osterloh and Frey

2000)• Incentives are effective for explicit knowledge, but they will crowd out intrinsic

motivations for sharing tacit knowledge

– Korean managers: social norm (positive impact) vs. extrinsic incentives (negative impact) (Bock et al., 2005)

• Intangible incentives increase contribution, fidelity, commitment, and sense of belonging in the information-sharing system

• In online systems, people anticipate feedback/recognition from others; in information sharing, this behavior is called “social approval and observational cooperation” (Cheshire 2007)

Goals

• Independent variables:– Tangible incentive: price, tips– Intangible incentives: rating, comments

• Comments before submitting a formal answer

• Dependent variable: # answers per answerer (GAR)

• H1: Both tangible and intangible incentives will have a positive effect on the number of answers provided by an answerer in the Google Answers Web site.

• H2: Tangible and intangible incentives will have different levels of contribution for frequent versus occasional answerers in predicting the number of answers provided

Results

Other potential hidden variables (that couldn’t be drawn from the dataset): personal gratification, sense of belonging to a community

Summary

• Askers value longer answers, quick responses, and reliable sources

• Answerers learn over time (as such, receiving higher ratings, slightly earning more)

• Answerers tend to specialize into several topics (which may lower actual hourly wage)

• Financial incentives improves quantity but not quality of answers (and reputation of repliers is related to quality)

• Tangible (monetary) and intangible incentives have positive impact on the number of answers per answerer