Aardvark shalini

AardvarkThe Anatomy of a Large-Scale Social Search

Engineby

Damon Horowitz and Sepandar D. Kamvar

Presented byShalini Sahoo

11/21/2011

Introduction

• Library vs village paradigm

• Traditional IR approaches follow the library paradigm

• In a village, information is passed from person to person

• The retrieval task consists of finding the right person (expert in that field)

• Example queries: (Pg 1) “Do you have any good baby-sitter recommendations in Palo-Alto for my 6-year-old twins? I’m looking for somebody that won’t let them watch TV.”

“Is it safe for me to take a cab alone at 3 am from SFO airport to my home in Berkeley?”

2

Differences

3

Library Village

Keywords are used to search Natural language used to ask questions

Short queries (~2.93 words) Verbose, highly contextualized and subjective (~18.6 words)

Knowledge base created by content publishers Community forms the knowledge base

Trust is based on authority Trust is based on intimacy

Retrieval involves finding the right document

Retrieval involves finding the right person

Aardvark

• Is* a social search engine based on the village paradigm

• It connects users live with friends or friends-of-friends who are able to answer their questions

• Users submit questions via Aardvark’s website, email, instant messenger or app on mobile devices

• It identifies and facilitates a live chat or email conversation with one or more topic experts in the users extended social network

• It was mainly used for asking subjective questions for which human judgement or recommendation was desired

4

* was - Aardvark was shut down on September 30th 2011

Aardvark

5

• It was originally developed by The Mechanical Zoo, a San-Francisco-based startup founded in 2007 by Max Ventilla, Damon Horowitz

Early 2008

A prototype version was launched

March 2009

It was released to public

February 2010

Google acquired itfor $50 million

September 2011

Google shut down Aardvark*

* A fall spring-clean: http://googleblog.blogspot.com/2011/09/fall-spring-clean.html

http://googleblog.blogspot.com/2011/09/fall-spring-clean.html

http://googleblog.blogspot.com/2011/09/fall-spring-clean.html

Outline

• Overview

• Anatomy

• Examples

• Analysis

• Evaluation

• Discussion

6

Outline➡ Overview‣ Main Components‣ The Initiation of a User‣ The Life of a Query

• Anatomy

• Examples

• Analysis

• Evaluation

• Discussion

7

Main Components

• Crawler and Indexer: To find and label resources that contain information

• Query Analyzer: To understand the user’s information need

• Ranking Function: To select the best resources to provide the information

• User Interface: To present the information to the user in an accessible and interactive form

8

The Initiation of a User• The first step involves forming the “Social Graph”

• Users can import contacts from:- social networking sites like Facebook or LinkedIn- webmail program like Gmail or Yahoo mail - invite friends to join

• Users in a common group or community (e.g. studied at UT Austin, Google summer interns 2011) are added to the social graph

• User’s topical expertise information is indexed:- Users can indicate the topics in which they have expertise- User’s friend can select topics for which they trust the user’s opinion- Users can indicate their personal webpages or blogs- User’s status updates from Facebook or Twitter (if available)

9

The Initiation of a User

• Forward Index: stores the userId, a scored list of topics, further scores about user behavior

• From this forward index, an inverted index is constructed

• Inverted Index: stores each topicId and a scored list of userIds (with expertise in that topic)

• Inverted index also stores scored list of userIds for features like answer quality and response time

10

The Life of a Query

11

IMEmailTwitteriPhoneWebSMS

Gateways

Tra

nsport

Layer

Msgs Conversation Manager

Question Analyzerclassifiers

Routing Engine

RSR

{RS}*

Index

DatabaseUsersGraphTopicsContent

Web ContentFacebookLinkedInEtc.

Importers

The Life of a Query

11


Gateways

Tra

nsport

Layer



Routing Engine

RSR

{RS}*

Index



Importers

The Life of a Query

11


Gateways

Tra

nsport

Layer



Routing Engine

RSR

{RS}*

Index



Importers

The Life of a Query

11


Gateways

Tra

nsport

Layer



Routing Engine

RSR

{RS}*

Index



Importers

The Life of a Query

11


Gateways

Tra

nsport

Layer



Routing Engine

RSR

{RS}*

Index



Importers

The Life of a Query

11


Gateways

Tra

nsport

Layer



Routing Engine

RSR

{RS}*

Index



Importers

The Life of a Query

11


Gateways

Tra

nsport

Layer



Routing Engine

RSR

{RS}*

Index



Importers

The Life of a Query

11


Gateways

Tra

nsport

Layer



Routing Engine

RSR

{RS}*

Index



Importers

The Life of a Query

11


Gateways

Tra

nsport

Layer



Routing Engine

RSR

{RS}*

Index



Importers

The Life of a Query

11


Gateways

Tra

nsport

Layer



Routing Engine

RSR

{RS}*

Index



Importers

Outline✓ Overview

➡ Anatomy‣ The Model‣ Social Crawling‣ Indexing People‣ Analyzing Questions‣ Ranking Algorithm‣ User Interface

• Examples

• Analysis

• Evaluation

• Discussion

12

The Model

• A network variant of an aspect model is used- It associates an unobserved class variable t ∊ T with each

observation (i.e., the successful answer of question q by user ui)

- It defines a query-independent probability of success for each potential asker/answerer pair (ui, uj)

• The scoring function s(ui, uj, q) is defined as a composition of the two probabilities

13

2. Query Analyzer. To understand the user’s informationneed (Section 3.4).

3. Ranking Function. To select the best resources to pro-vide the information (Section 3.5).

4. UI. To present the information to the user in an ac-cessible and interactive form (Section 3.6).

Most corpus-based search engines have similar key com-ponents with similar aims [4], but the means of achievingthose aims are quite di↵erent.

Before discussing the anatomy of Aardvark in depth, it isuseful to describe what happens behind the scenes when anew user joins Aardvark and when a user asks a question.

2.2 The Initiation of a UserWhen a new user first joins Aardvark, the Aardvark sys-

tem performs a number of indexing steps in order to be ableto direct the appropriate questions to her for answering.

Because questions in Aardvark are routed to the user’s ex-tended network, the first step involves indexing friendshipand a�liation information. The data structure responsiblefor this is the Social Graph. Aardvark’s aim is not to builda social network, but rather to allow people to make use oftheir existing social networks. As such, in the sign-up pro-cess, a new user has the option of connecting to a social net-work such as Facebook or LinkedIn, importing their contactlists from a webmail program, or manually inviting friendsto join. Additionally, anybody whom the user invites tojoin Aardvark is appended to their Social Graph – and suchinvitations are a major source of new users. Finally, Aard-vark users are connected through common “groups” whichreflect real-world a�liations they have, such as the schoolsthey have attended and the companies they have workedat; these groups can be imported automatically from socialnetworks, or manually created by users. Aardvark indexesthis information and stores it in the Social Graph, which isa fixed width ISAM index sorted by userId.

Simultaneously, Aardvark indexes the topics about whichthe new user has some level of knowledge or experience.This topical expertise can be garnered from several sources:a user can indicate topics in which he believes himself tohave expertise; a user’s friends can indicate which topicsthey trust the user’s opinions about; a user can specify anexisting structured profile page from which the Topic Parser

parses additional topics; a user can specify an account onwhich they regularly post status updates (e.g., Twitter orFacebook), from which the Topic Extractor extracts top-ics (from unstructured text) in an ongoing basis (see Sec-tion 3.3 for more discussion); and finally, Aardvark observesthe user’s behavior on Aardvark, in answering (or electingnot to answer) questions about particular topics.

The set of topics associated with a user is recorded inthe Forward Index, which stores each userId, a scored list oftopics, and a series of further scores about a user’s behavior(e.g., responsiveness or answer quality). From the ForwardIndex, Aardvark constructs an Inverted Index. The InvertedIndex stores each topicId and a scored list of userIds thathave expertise in that topic. In addition to topics, the In-verted Index stores scored lists of userIds for features likeanswer quality and response time.

Once the Inverted Index and Social Graph for a user arecreated, the user is now active on the system and ready toask her first question.


Gateways

Tra

nsport

Layer



Routing Engine

RSR

{RS}*

Index



Importers

Figure 1: Schematic of the architecture of Aardvark

2.3 The Life of a QueryA user begins by asking a question, most commonly through

instant message or text message The question gets sent fromthe input device to the Transport Layer, where it is normal-ized to a Message data structure, and sent to the Conversa-

tion Manager. Once the Conversation Manager determinesthat the message is a question, it sends the question to theQuestion Analyzer to determine the appropriate topics forthe question. The Conversation Manager informs the askerwhich primary topic was determined for the question, andgives the asker the opportunity to edit it. It simultaneouslyissues a Routing Suggestion Request to the Routing Engine.The routing engine plays a role analogous to the rankingfunction in a corpus-based search engine. It accesses theInverted Index and Social Graph for a list of candidate an-swerers, and ranks them to reflect how well it believes theycan answer the question, and how good of a match theyare for the asker. The Routing Engine returns a ranked listof Routing Suggestions to the Conversation Manager, whichthen contacts the potential answerers — one by one, or afew at a time, depending upon a Routing Policy — and asksthem if they would like to answer the question, until a sat-isfactory answer is found. The Conversation Manager thenforwards this answer along to the asker, and allows the askerand answerer to exchange followup messages.

3. ANATOMY

3.1 The ModelThe core of Aardvark is a statistical model for routing

questions to potential answerers. We use a network variantof what has been called an aspect model [12], that has twoprimary features. First, it associates an unobserved classvariable t 2 T with each observation (i.e., the successfulanswer of question q by user u

i

). In other words, the proba-bility p(u

i

|q) that user i will successfully answer question qdepends on whether q is about the topics t in which u

i

hasexpertise1:

p(ui

|q) =X

t2T

p(ui

|t)p(t|q) (1)

1Equation 1 is a simplification of what Aardvark actuallyuses to match queries to answerers, but we present it thisway for clarity and conciseness.

The second main feature of the model is that it definesa query-independent probability of success for each poten-tial asker/answerer pair (u

i

, uj

), based upon their degree ofsocial connectedness and profile similarity. In other words,we define a probability p(u

i

|uj

) that user ui

will deliver asatisfying answer to user u

j

, regardless of the question.We then define the scoring function s(u

i

, uj

, q) as the com-position of the two probabilities.

s(ui

, uj

, q) = p(ui

|uj

) · p(ui

|q) = p(ui

|uj

)X

t2T

p(ui

|t)p(t|q)

(2)Our goal in the ranking problem is: given a question q

from user uj

, return a ranked list of users ui

2 U thatmaximizes s(u

i

, uj

, q).Note that the scoring function is composed of a query-

dependent relevance score p(ui

|q) and a query-independentquality score p(u

i

|uj

). This bears similarity to the rankingfunctions of traditional corpus-based search engines such asGoogle [4]. The di↵erence is that unlike quality scores likePageRank [18], Aardvark’s quality score aims to measureintimacy rather than authority. And unlike the relevancescores in corpus-based search engines, Aardvark’s relevancescore aims to measure a user’s potential to answer a query,rather than a document’s existing capability to answer aquery.

Computationally, this scoring function has a number ofadvantages. It allows real-time routing because it pushesmuch of the computation o✏ine. The only component prob-ability that needs to be computed at query time is p(t|q).Computing p(t|q) is equivalent to assigning topics to a ques-tion — in Aardvark we do this by running a probabilisticclassifier on the question at query time (see Section 3.4).The distribution p(u

i

|t) assigns users to topics, and the dis-tribution p(u

i

|uj

) defines the Aardvark Social Graph. Bothof these are computed by the Indexer at signup time, andthen updated continuously in the background as users an-swer questions and get feedback (see Section 3.3). The com-ponent multiplications and sorting are also done at querytime, but these are easily parallelizable, as the index issharded by user.

3.2 Social CrawlingA comprehensive knowledge base is important for search

engines as query distributions tend to have a long tail [13].In corpus-based search engines, this is achieved by large-scale crawlers and thoughtful crawl policies. In Aardvark,the knowledge base consists of people rather than docu-ments, so the methods for acquiring and expanding a com-prehensive knowledge base are quite di↵erent.

With Aardvark, the more active users there are, the morepotential answerers there are, and therefore the more com-prehensive the coverage. More importantly, because Aard-vark looks for answerers primarily within a user’s extendedsocial network, the denser the network, the larger the e↵ec-tive knowledge base.

This suggests that the strategy for increasing the knowl-edge base of Aardvark crucially involves creating a goodexperience for users so that they remain active and are in-clined to invite their friends. An extended discussion of thisis outside of the scope of this paper; we mention it here onlyto emphasize the di↵erence in the nature of “crawling” insocial search versus traditional search.

Given a set of active users on Aardvark, the e↵ective

breadth of the Aardvark knowledge base depends upon de-signing interfaces and algorithms that can collect and learnan extended topic list for each user over time, as discussedin the next section.

3.3 Indexing PeopleThe central technical challenge in Aardvark is selecting

the right user to answer a given question from another user.In order to do this, the two main things Aardvark needsto learn about each user u

i

are: (1) the topics t he mightbe able to answer questions about p

smoothed

(t|ui

); (2) theusers u

j

to whom he is connected p(ui

|uj

).Topics. Aardvark computes the distribution p(t|u

i

) oftopics known by user u

i

from the following sources of infor-mation:

• Users are prompted to provide at least three topicswhich they believe they have expertise about.

• Friends of a user (and the person who invited a user)are encouraged to provide a few topics that they trustthe user’s opinion about.

• Aardvark parses out topics from users’ existing onlineprofiles (e.g., Facebook profile pages, if provided). Forsuch pages with a known structure, a simple TopicParsing algorithm uses regular expressions which weremanually devised for specific fields in the pages, basedupon their performance on test data.

• Aardvark automatically extracts topics from unstruc-tured text on users’ existing online homepages or blogsif provided. For unstructured text, a linear SVM iden-tifies the general subject area of the text, while anad-hoc named entity extractor is run to extract morespecific topics, scaled by a variant tf-idf score.

• Aardvark automatically extracts topics from users’ sta-tus message updates (e.g., Twitter messages, Facebooknews feed items, IM status messages, etc.) and fromthe messages they send to other users on Aardvark.

The motivation for using these latter sources of profiletopic information is a simple one: if you want to be ableto predict what kind of content a user will generate (i.e.,p(t|u

i

)), first examine the content they have generated inthe past. In this spirit, Aardvark uses web content not as asource of existing answers about a topic, but rather, as anindicator of the topics about which a user is likely able togive new answers on demand.

In essence, this involves modeling a user as a content-generator, with probabilities indicating the likelihood shewill likely respond to questions about given topics. Eachtopic in a user profile has an associated score, dependingupon the confidence appropriate to the source of the topic.In addition, Aardvark learns over time which topics not tosend a user questions about by keeping track of cases whenthe user: (1) explicitly “mutes” a topic; (2) declines to an-swer questions about a topic when given the opportunity;(3) receives negative feedback on his answer about the topicfrom another user.

Periodically, Aardvark will run a topic strengthening algo-rithm, the essential idea of which is: if a user has expertisein a topic and most of his friends also have some exper-tise in that topic, we have more confidence in that user’s

The Model

• The goal of the ranking algorithm is: given a question q from user uj, return a ranked list of users ui ∊ U that maximizes the scoring function s(ui, uj, q)

• The scoring function allows real-time routing because much of the computation is done offline

• The only term which needs to be computed at query-time is p(t|q)

• The distribution p(ui|t) assigns users to topics, and the distribution p(ui|uj) defines the Aardvark social graph, both of these are computed by the Indexer at signup time

14

Social Crawling

• In Aardvark, people form the knowledge base rather than documents

• The more active users there are, the more potential answerers

• So it is important for Aardvark to create a good experience for users so that they remain active and inclined to invite their friends

• The breadth of the Aardvark knowledge base depends upon designing interfaces and algorithms to update the topic lists for each user over time

15

Indexing People

• The distribution (p(t|uj)) of topics known by user ui is computed from the following sources:

- Users can indicate the topics in which they have expertise- User’s friend can select topics for which they trust the user’s opinion- Users can indicate their personal webpages or blogs- User’s status updates from Facebook or Twitter (if available)

• Over time, Aardvark learns which topics not to send to a particular user by keeping track of:

- when user explicitly “mutes” a topic- declines to answer questions about a topic when given the chance- receives negative feedback on his answer from the asker

16

Indexing People

• Periodically, a topic strengthening algorithm is used

• For a user ui and his group of friends U, and for some topic t, if p(t|ui) ≠ 0, then

s(t|ui) = p(t|ui) + n∑u ∊ U p(t|u)

where n is a small constant

• Other smoothing techniques are used to record the possibility that a user might be able to answer about additional topics not explicitly mentioned in their profile

• p(ui|t) is computed using Bayes’ law

17

level of expertise than if he were alone in his group withknowledge in that area. Mathematically, for some user u

i

,his group of friends U , and some topic t, if p(t|u

i

) 6= 0,then s(t|u

i

) = p(t|ui

) + �P

u2U

p(t|u), where � is a smallconstant. The s values are then renormalized to form prob-abilities.

Aardvark then runs two smoothing algorithms the pur-pose of which are to record the possibility that the user maybe able to answer questions about additional topics not ex-plicitly recorded in her profile. The first uses basic collabo-rative filtering techniques on topics (i.e., based on users withsimilar topics), the second uses semantic similarity2.

Once all of these bootstrap, extraction, and smoothingmethods are applied, we have a list of topics and scoresfor a given user. Normalizing these topic scores so thatP

t2T

p(t|ui

) = 1, we have a probability distribution fortopics known by user u

i

. Using Bayes’ Law, we compute foreach topic and user:

p(ui

|t) =p(t|u

i

)p(ui

)p(t)

, (3)

using a uniform distribution for p(ui

) and observed topicfrequencies for p(t). Aardvark collects these probabilitiesp(u

i

|t) indexed by topic into the Inverted Index, which al-lows for easy lookup when a question comes in.

Connections. Aardvark computes the connectedness be-tween users p(u

i

|uj

) in a number of ways. While socialproximity is very important here, we also take into accountsimilarities in demographics and behavior. The factors con-sidered here include:

• Social connection (common friends and a�liations)

• Demographic similarity

• Profile similarity (e.g., common favorite movies)

• Vocabulary match (e.g., IM shortcuts)

• Chattiness match (frequency of follow-up messages)

• Verbosity match (the average length of messages)

• Politeness match (e.g., use of “Thanks!”)

• Speed match (responsiveness to other users)

Connection strengths between people are computed using aweighted cosine similarity over this feature set, normalizedso that

Pui2U

p(ui

|uj

) = 1, and stored in the Social Graphfor quick access at query time.

Both the distributions p(ui

|uj

) in the Social Graph andp(t|u

i

) in the Inverted Index are continuously updated asusers interact with one another on Aardvark.

3.4 Analyzing QuestionsThe purpose of the Question Analyzer is to determine a

scored list of topics p(t|q) for each question q representingthe semantic subject matter of the question. This is the onlyprobability distribution in equation 2 that is computed atquery time.

2In both the Person Indexing and the Question Analysiscomponents, “semantic similarity” is computed by usingan approximation of distributional similarity computed overWikipedia and other corpora; this serves as a proxy measureof the topics’ semantic relatedness.

It is important to note that in a social search system, therequirement for a Question Analyzer is only to be able tounderstand the query su�ciently for routing it to a likelyanswerer. This is a considerably simpler task than the chal-lenge facing an ideal web search engine, which must attemptto determine exactly what piece of information the user isseeking (i.e., given that the searcher must translate her infor-mation need into search keywords), and to evaluate whethera given web page contains that piece of information. Bycontrast, in a social search system, it is the human answererwho has the responsibility for determining the relevance ofan answer to a quesion — and that is a function which hu-man intelligence is extremely well-suited to perform! Theasker can express his information need in natural language,and the human answerer can simply use her natural un-derstanding of the language of the question, of its tone ofvoice, sense of urgency, sophistication or formality, and soforth, to determine what information is suitable to includein a response. Thus, the role of the Question Analyzer ina social search system is simply to learn enough about thequestion that it may be sent to appropriately interested andknowledgeable human answerers.

As a first step, the following classifiers are run on eachquestion: A NonQuestionClassifier determines if the inputis not actually a question (e.g., is it a misdirected message, asequence of keywords, etc.); if so, the user is asked to submita new question. An InappropriateQuestionClassifier deter-mines if the input is obscene, commercial spam, or otherwiseinappropriate content for a public question-answering com-munity; if so, the user is warned and asked to submit a newquestion. A TrivialQuestionClassifier determines if the in-put is a simple factual question which can be easily answeredby existing common services (e.g., “What time is it now?”,“What is the weather?”, etc.); if so, the user is o↵ered anautomatically generated answer resulting from traditionalweb search. A LocationSensitiveClassifier determines if theinput is a question which requires knowledge of a particularlocation, usually in addition to specific topical knowledge(e.g., “What’s a great sushi restaurant in Austin, TX?”); ifso, the relevant location is determined and passed along tothe Routing Engine with the question.

Next, the list of topics relevant to a question is producedby merging the output of several distinct TopicMapper algo-rithms, each of which suggests its own scored list of topics:

• A KeywordMatchTopicMapper passes any terms in thequestion which are string matches with user profiletopics through a classifier which is trained to deter-mine whether a given match is likely to be semanticallysignificant or misleading.3

• A TaxonomyTopicMapper classifies the question textinto a taxonomy of roughly 3000 popular question top-ics, using an SVM trained on an annotated corpus ofseveral millions questions.

• A SalientTermTopicMapper extracts salient phrases fromthe question — using a noun-phrase chunker and a tf-

3For example, if the string “camel wrestling” occurs in aquestion, it is likely to be semantically relevant to a userwho has “camel wrestling” as a profile topic; whereas thestring “running” is too ambiguous to use in this mannerwithout further validation, since it might errantly route aquestion about “running a business” to a user who knowsabout fitness.

Connectedness

• Connectedness between users p(ui|uj) is computed using a weighted cosine similarity over the following feature set:

- Social connection - Demographic similarity- Profile similarity - Vocabulary match- Chattiness match- Verbosity match - Politeness match- Speed match

• p(ui|uj) is stored in the social graph

18

Analyzing Questions

• The main goal of the Question Analyzer is: given a question q , determine a scored list of topics p(t|q) for each question

• The following classifiers are run on a question:- NonQuestion Classifier- InappropriateQuestion Classifier- TrivialQuestion Classifier- LocationSensitive Classifier

• Next, the list of relevant topics is produced by merging outputs from several TopicMapper algorithms

- KeywordMatchTopicMapper- TaxonomyTopicMapper- SalientTermTopicMapper- UserTagTopicMapper

19

Analyzing Questions

• The TopicMapper algorithms are continuously evaluated

• Given a question all the returned topics to select an answerer, and a much larger list of relevant topics are assigned scores by two human judges

• 89% precision and 84% recall of relevant topics

20

Ranking Algorithm

• The topic list generated by the Question Analyzer is sent to the Routing Engine which then determines the top answerers for the given question

• The main factors that determines the ranking of users are:- Topic expertise p(ui|q)- Connectedness p(ui|uj)- Availability

• From this ordered list of users the Routing Engine then filters out users who should not be contacted

- based on preferred time of contact- based on the frequency of times they have been contacted in the

recent past

21

User Interface

• The various user interfaces of Aardvark are built on top of the real time communication channels such as IM, email, SMS, iPhone, Twitter and Web-based messaging

22

User Interface

23

User Interface

24

Outline

✓ Overview

✓ Anatomy

➡ Examples

• Analysis

• Evaluation

• Discussion

25

Examples

26

EXAMPLE 1

(Reply from Nick to Mark)you're very welcome. hope the days they're open for lunch work...

(+4 minutes -- Answer from Nick T./28/M/SanFrancisco,CA -- a friend of your friend Fritz Schwartz)fringale (fringalesf.com) in soma is a good bet; small, fancy, french (the french actually hang out there too). Lunch: Tuesday - Friday: 11:30am - 2:30pm

(Question from Mark C./M/LosAltos,CA)I am looking for a restaurant in San Francisco that is open for lunch. Must be very high-end and fancy (this is for a small, formal, post-wedding gathering of about 8 people).

(Reply from Mark to Nick)Thanks Nick, you are the best PM ever!

EXAMPLE 2

(+1 hour -- Answer from Fred M./29/M/Marina,SF)Quince is a little fancy... La Mar is pretty fantastic for cevice - like the Slanted Door of peruvian food...

(+7 minutes -- Answer from Paul D./M/SanFrancisco,CA -- A friend of your friend Sebastian V.)For business dinner I enjoyed Kokkari Estiatorio at 200 Jackson. If you prefer a place in SOMA i recommend Ozumo (a great sushi restaurant).

(Question from James R./M/TwinPeaksWest,SF)What is the best new restaurant in San Francisco for a Monday business dinner? Fish & Farm? Gitane? Quince (a little older)?

(Reply from James to Paul) thx I like them both a lot but I am ready to try something new

EXAMPLE 3

(+10 minutes -- Answer from Bob F./M/Mission,SF -- you are connected through Mathias' friend Samantha S.) Cool question. Spork is usually my top choice for a first date, because in addition to having great food and good really friendly service, it has an atmosphere that's perfectly in between casual and romantic. It's a quirky place, interesting funny menu, but not exactly non-traditional in the sense that you're not eating while suspended from the ceiling or anything

(Reply from Brian to Anthony) Tommy as in the Who's rock opera? COOL!

(+6 minutes -- Answer from Anthony D./M/Sunnyvale,CA -- you are both in the Google group) Take her to the ROTL production of Tommy, in the Mission. Best show i've seen all year!

(+4 minutes -- Answer from Dan G./M/SanFrancisco,CA)Start with drinks at NocNoc (cheap, beer/wine only) and then dinner at RNM (expensive, across the street).

(Question from Brian T./22/M/Castro,SF) What is a good place to take a spunky, off-the-cuff, social, and pretty girl for a nontraditional, fun, memorable dinner date in San Francisco?

(Reply from Brian to Dan) Thanks!

Examples

27

Examples

28

Outline

✓ Overview

✓ Anatomy

✓ Examples

➡ Analysis

• Evaluation

• Discussion

29

Analysis

• As of October 2009, Aardvark had 90361 registered users

• The average query volume was 3167.2 questions per day in this period

30

Users

Analysis• Mobile users were particularly active

- It is easier to reply to questions in the form of IM or SMS on phone- People are comfortable using natural language in an IM setting

rather than in a web search setting

• Questions are highly contextualized

- Average query length is 18.6 words

• Questions often have a subjective element

31restaurants & bars

miscellaneous

websites & internet appsmusic, movies, TV, books

technology & programming

local services

product reviews & help

Aardvark

travel

restaurants & bars

finance & investinghome & cookingsports & recreationbusiness research

Analysis• Questions get answered quickly

• Answers are of high quality- Answers are comprehensive and concise- Median answer length was 22.2 words- 70.4% of inline feedback rated answers as ‘good’, 14.1% rated

answers as ‘OK’ and 15.5% were rated as ‘bad’

32

Users

Figure 7: Aardvark user growth

asker; the second answer came from a coworker; and thethird answer came from a friend-of-friend-of-friend. Thethird answer, which is the most detailed, came from a userwho has topics in his profile related to both “restaurants”and “dating”.

One of the most interesting features of Aardvark is thatit allows askers to get answers that are hypercustomized totheir information need. Very di↵erent restaurant recommen-dations are appropriate for a date with a spunky and spon-taneous young woman, a post-wedding small formal familygathering, and a Monday evening business meeting — andhuman answerers are able to recognize these constraints. Itis also interesting to note that in most of these examples (asin the majority of Aardvark questions), the asker took thetime to thank the answerer for helping out.

5. ANALYSISThe following statistics give a picture of the current usage

and performance of Aardvark.Aardvark was first made available semi-publicly in a beta

release in March of 2009. From March 1, 2009 to October20, 2009, the number of users grew to 90,361, having askeda total of 225,047 questions and given 386,702 answers. Allof the statistics below are taken from the last month of thisperiod (9/20/2009-10/20/2009).

Aardvark is actively used As of October, 2009, 90,361users have created accounts on Aardvark, growing or-ganically from 2,272 users since March 2009. In thisperiod, 50,526 users (55.9% of the user base) generatedcontent on Aardvark (i.e., asked or answered a ques-tion), while 66,658 users (73.8% of the user base) pas-sively engaged (i.e., either referred or tagged other peo-ples questions). The average query volume was 3,167.2questions per day in this period, and the median activeuser issued 3.1 queries per month. Figure 7 shows thenumber of users per month from early testing throughOctober 2009.

Mobile users are particularly active Mobile users hadan average of 3.6322 sessions per month, which is sur-prising on two levels. First, mobile users of Aardvarkare more active than desktop users. (As a point ofcomparison, on Google, desktop users are almost 3

restaurants & bars

miscellaneous

websites & internet appsmusic, movies, TV, books

technology & programming

local services

product reviews & help

Aardvark

travel

restaurants & bars

finance & investinghome & cookingsports & recreationbusiness research

Figure 8: Categories of questions sent to Aardvark

0−3 min 3−6 min 6−12 min 12−30 min30min−1hr 1−4 hr 4+ hr0

0.5

1

1.5

2

2.5 x 104Q

uest

ions

Ans

wer

ed

Figure 9: Distribution of questions and answering

times.

times as active as mobile users [14].) Second, mo-bile users of Aardvark are almost as active in absoluteterms as mobile users of Google (who have on average5.68 mobile sessions per month [14]). This is quite sur-prising for a service that has only been available for 6months.

We believe this is for two reasons. First, browsingthrough traditional web search results on a phone isunwieldy. On a phone, it’s more useful to get a sin-gle short answer that’s crafted exactly to your query.Second, people are used to using natural language withphones, and so Aardvark’s query model feels natural inthat context. These considerations (and early exper-iments) also suggest that Aardvark mobile users willbe similarly active with voice-based search.

Questions are highly contextualized As compared to websearch, where the average query length is between 2.2– 2.9 words [14, 19], with Aardvark, the average querylength is 18.6 words (median=13). While some of thisincreased length is due to the increased usage of func-tion words, 45.3% of these words are content wordsthat give context to the query. In other words, ascompared to traditional web search, Aardvark ques-tions have 3–4 times as much context.

The addition of context results in a greater diversity ofqueries. While in Web search, between 57 and 63% ofqueries are unique [19, 20], in Aardvark 98.1% of ques-tions are unique (and 98.2% of answers are unique).

Questions often have a subjective element A manualtally of 1000 random questions between March and Oc-tober of 2009 shows that 64.7% of queries have a sub-jective element to them (for example, “Do you know ofany great delis in Baltimore, MD?” or “What are thethings/crafts/toys your children have made that made

Analysis

• There are a broad range of answerers

• Social proximity matters

• People are indexable

33

Outline

✓ Overview

✓ Anatomy

✓ Examples

✓ Analysis

➡ Evaluation

• Discussion

34

Evaluation

• Compared to Google!

• “Do you want to help Aardvark run an experiment?” was inserted into a random sample of active questions

• Users were asked to reformulate their question as a query and search on Google

• Users time how long it took to find a satisfactory result and also rate the quality of answers

• 71.5% on Aardvark, with a mean rating of 3.93

• 70.5% on Google, with a mean rating of 3.07

35

Outline

✓ Overview

✓ Anatomy

✓ Examples

✓ Analysis

✓ Evaluation

➡ Discussion

36

Discussion• Participation Fatigue: (Pg 9) “86.7% users have been

contacted by Aardvark with a request to answer a question, and of those, 70% have looked at the question and 38% could answer a question. 20% of the users accounted for 85% of answers” What happens when this thin slice of users get overwhelmed and start dropping out?

• Availability: There can be cases when the topic expert(s) in your social graph might not be online. Do you think having an “offline” mode be helpful?

• Evaluation: Can we get a better understanding of how well Aardvark worked had it been compared to another social search engine which works on the same paradigm? How can that be achieved?

37

Technology

Aardvark shalini